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DEFINITION  OF  SYSTEM  EFFECTIVENESS 

SYSTEM  EFFECTIVENESS  —  A 
MEASURE  OF  THE  EXTENT  TO  WHICH 
A  SYSTEM  CAN  BE  EXPECTED  TO 
COMPLETE  ITS  ASSIGNED  MISSION 
WITHIN  AN  ESTABLISHED  TIME 
FRAME  UNDER  STATED  ENVIRONMENTAL 
CONDITIONS. 
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The  main  purpose  of  this  pamphlet  is  to  provide  to  personnel  of  the 
Naval  Material  Support  Establishment  (NMSE)  a  collection  of  papers  in 
a  single  volume  which  reflect  the  attitude  and  philosophy  of  the  Chief 
of  Nava.1.  Material  tcwfnds  various  aspects  of  systems  effectiveness.  It 
also  provides  a  discussion  of  the  planning,  design,  and  cost  considera¬ 
tions  in  system  development  as  well  as  some  techniques  now  being  utilized 
in  the  NMSE  ih  order  to  realize  the  development  of  effective  systems. 

The  majority  of  the  papers  included  in  this  pamphlet  were  presented 
by  the  Chief  of  Naval  Material  and  his  representatives  at  the  North¬ 
eastern  States,  Navy  Research  and  Development  Clinic  held  18-20  Nov 
1964  in  Philadelphia.  Several  other  papers  which  were  presented  to 
ottK~  audiences  were  also  considered  appropriate  for  inclusion  in  this 
pamphlet.. 

This  publication  has  been  reviewed  and  approved  in  compliance  with 
SEC'IAVINST  -5600.16. 


Fear  Admiral  E.  A.  Ruckner,  USN 
Deputy  Chief  of  Naval  Material  for 
Development/Chief  of  Naval  Development 
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To  this  point  I  have  been  suggesting  specific  areas  of  effort, 
each  of  which  would  be  of  interest  to  certain  of  you  in  attendance  here. 
I  would  like  now  to  focus  your  attention  very  briefly  on  an  area  of 
effort  in  which  all  of  you  can  participate.  In  fact  —  it  is  an  area 
in  which  everyone  participating  in  Navy  R&D  contracts  will  be  involved. 

I  speak  of  the  area  which  we  term  Systens  Effectiveness. 

In  this  era  of  complex  combinations  of  men  and  machines ,  Systems 
Effectiveness,  and  its  fiscal  corollary.  Cost  Effectiveness,  constitute 
the  most  important  area  of  concern  to  military  R&D  Management.  It  is, 
or  soon  will  be,  of  equal  import  to  civilian  R&D  Management  which 
addresses  itself  to  military  systems. 

What  does  this  mean  to  you? 

It  means  that  Systems  Effectiveness,  which  we  define  as  a  measure 
of  the  extent  to  which  a  system  can  be  expected  to  complete  its  assigned 
mission  within  an  established  time  frame  under  stated  environmental 
conditions,  is  the  focus  of  our  research  and  development  efforts.  It 
means  that  Systems  Effectiveness  is  the  measure  of  the  goodness  of  our 
systems.  Systems  Effectiveness  is  thus  a  matter  of  paramount  concern 
to  us,  and  to  you. 

The  manager,  the  scientist  and  the  engineer,  working  toward  the 
development  of  a  system,  must  take  into  account  all  of  the  attributes 
of  the  system  which  we  refer  to  as  qualitative  characteristics.  Such 
factors  as  reliability,  maintainability,  operability,  logistic  support- 
ability,  human  factors  and  all  the  other  factors  affecting  the  goodness 
of  a  system,  must  be  thoroughly  considered  in  development  planning. 


It  is  quite  true  that  each  of  the  characteristics  which  I  have 
mentioned  have  been  considered  to  varying  degrees  as  factors  in 
existing  weapon  systems.  What  is  needed  is  correlated  consideration 
of  all  of  these  characteristics  in  systems  design.  Only  by  providing 
in  the  early  developmental  stages  for  reliability,  maintainability, 
simplicity,  supportability  and  similar  factors  can  the  systems  you 
develop  bear  up  under  Systems  Effectiveness  analysis. 

lour  participation  in  this  effort  will,  I  believe,  be  in  two 
separate  areas.  The  first  will  be  the  further  detailed  development 
of  Systems  Effectiveness  methodology,  techniques  and  model  structures. 
The  second  will  be  the  application  of  this  discipline  to  your  proposals. 

We  can  no  longer  afford  the  "build  one  and  try  it"  approach  with 
a  subsequent  "get  well"  effort  to  patch  on  reliability,  maintainability, 
value  engineering  and  the  like.  We  cannot  afford  to  develop  systems 
using  men  as  multi-purpose  gap-fillers  between  machine  interfaces. 
Neither  can  we  accept  weapons  systems  which  must  be  staffed  by  crews 
of  PHD’s.  We  must  develop  mathematical  modeling  techniques  with  which 
to  do  our  systems  engineering  homework.  These  models  cannot  be  achieved 
without  a  cohesive  discipline  within  which  they  can  be  structured. 

This  discipline  we  term  Systems  Effectiveness.  In  the  highly  complex 
weaponry  of  modern  warfare,  this  discipline  is  absolutely  necessary. 

There  is  one  additional  thought  I  would  like  to  leave  with  ycu. 

In  reading  newly  issued  Specific  Operational  Requirements,  I  have 
repeatedly  observed  a  much  needed  and  increasing  emphasis  on  the 
position  a<vi  role  of  human  engineering  in  the  new  weapons  systems. 


These  systems  are  often  thought  of,  and  rightly,  as  combined  man- 
machine  systems.  The  Navy  will  continue  to  be  composed  both  of 
machines  and  of  people.  It  is  increasingly  necessary  that  the  weapons 
and  support  system  of  the  future  properly  blend  the  capabilities  and  the 
limitations  of  the  man  with  those  of  the  machine.  The  day  is  past  when 
the  man  can  be  regarded  as  an  entity  apart  from  the  machine.  He  is 
explicitly  a  part  of  the  weapons  syrtsm,  and  he  contributes  to  its 
effectiveness  those  capabilities  which  are  uniquely  his.  Systems  ef¬ 
fectiveness  must  take  into  account  the  man  who  operates  the  system,  and 
the  personal  reactions  of  people.  Of  all  the  scientific  contributions 
yet  to  be  made  to  the  defense  of  the  country,  it  is  probable  that  among 
the  most  valuable  will  be  contributions  from  the  life  sciences, 
particularly  the  behavioral  science  groups. 

In  conclusion,  I’m  sure  you  will  agree  that  it  will  be  interesting 
to  see,  in  the  decade  following  1975,  how  closely  our  prognostications 
fit  reality.  But  whatever  the  future  holds  it  is  certain  that  the  Navy 
will  remain  an  important  national  security  force,  its  roles  and  missions 
relatively  unchanged,  but  its  weapons  and  tactical  methods  greatly 
affected  by  burgeoning  technologies. 

It  has  been  said  that  the  wars  of  the  future  will  be  won  in  the 
laboratory.  I  suspect  that  the  security  of  our  country  ten  to  twenty 
years  from  now  will  depend  in  large  part  on  the  ability  of  all  of  us 
here  today  —  military  and  civilian  —  to  master  the  expanding 
technologies  which  can  ultimately  spell  triumph  or  tragedy  for  the 
United  States  and  the  free  world. 


(excerpt) 
by  LeRoy  Rosen 

Program  Manager  of  FRISCO  in 
Bureau  of  Ships 


Die  organization  required  to  handle  a  program  that  encompasses 
so  many  disciplines  as  does  FRISCO  is  a  special  one.  Figure  1  shows 
the  various  functions  that  must  he  coordinated  under  the  Program 
Office.  Ho  single  contractor  or  Navy  laboratory  has  ever  performed 
the  entire  design  of  a  submarine  tactical  control  system.  Therefore 
it  has  been  necessary  to  organize  the  Havy  laboratories  to  work  on 
this  task,  with  each  participating  in  areas  concerning  its  own 
specialty.  These  laboratories  are  dispersed  around  the  country,  so 
that  good  coantunl cations  among  them  and  with  the  Program  Office  is 
extremely  Important  in  achieving  the  project  objectives.  Hence,  a 
special  set  of  tools  is  required  to  perform  a  system  integration  in 
which  various  parts  of  the  system  are  being  developed  at  geographically 
dispersed  locations.  To  FRISCO 's  knowledge,  there  were  no  tools 
available  in  industry  that  could  be  utilized  for  technical  management, 
m  is  a  useful  tool  for  scheduling  and  estimating  costs  for  a  system 
but  the  main  concern  of  FRISCO  are  tools  to  insure  the  technical  goals 
of  a  system  and  it  was  these  which  were  lacking. 

The  approach  tfc  t  FRISCO  derived  to  meet  this  problem  requires 
the  accompanying  series  of  steps.  The  tasks  shown  above  the  dotted 
line,  in  Figure  2,  are  the  responsibility  of  the  Program  Office. 

Those  below  the  line  are  the  responsibility  of  the  laboratories  and/or 
contractors. 

Through  these  procedures,  the  Impact  of  the  given  threats  and  mis¬ 
sions  of  the  nuoiear  attack  submarine  is  determined.  The  information 
on  the  threat  and  mission  le  obtained  from  the  appropriate  sections  of 
the  levy  but  this  moat  be  translated  into  technological  requirements 
for  men  and  equipments. 

The  entire  environment  in  which  a  submarine  can  be  expected  to 


operate  is  far  too  complex  to  study  exhaustively.  Therefore,  a 
number  of  tactical  models  have  been  postulated  to  represent  various 
elements  of  the  complete  environment.  It  is  expected  that  by  study¬ 
ing  each  of  these  models  in  detail  and  then  summing  over  the  require¬ 
ments  generated  by  each,  the  overall  system  design  goals  can  be 
identified.  By  an  appropriate  selection  of  models,  a  reasonable 
picture  of  the  entire  system  should  be  obtained. 

There  are  seven  variables  that  must  be  considered  in  formu¬ 
lating  a  tactical  model  (Figure  3).  These  are:  mission,  area,  force 
mix,  political  situation,  physical  situation,  technological  assump¬ 
tions,  and  tactics. 

"Missions*  include  such  types  of  operations  as  AStf,  minelaying, 
and  the  sinking  of  surface  ships.  "Area"  takes  into  account  the  geo¬ 
graphy  of  the  location  in  which  the  operation  occurs  and  identifies 
restricted  or  unrestricted  areas,  deep  or  shallow  water,  etc.  The 
"force  mix"  is  stated  for  both  own  and  enemy  ship,  indicating  whether 
either  is  alone  or  operating  in  combination  with  other  submarines, 
surface  ships  or  aircraft.  The  "political  situation"  —  cold  war, 
nuclear,  non-nuclear  war  —  is  important  as  it  is  reflected  in  the 
tactics  that  the  submarine  and  its  target  employs.  The  "physical 
situation"  points  out  the  physical  details  of  the  chosen  area.  This 
includes  water  temperature,  current,  salinity,  thermal  layers,  etc. 

It  is  important  to  choose  an  actual  location  where  such  data  are 
available  to  insure  that  all  the  people  working  on  the  program  are 
using  the  same  set  of  conditions.  The  "technological  assumptions" 
describe  what  the  enemy's  capabilities  are  anticipated  to  be  in  the 
time  scale  under  consideration.  The  last  variable,  "tactics",  involves 
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a  new  concept  since  at  the  present,  tactics  are  generated  from  the 
optimal  usage  of  existing  equipments.  FRISCO  is  dealing  with  new 
equipments;  it  should  therefore  be  recognized  that  tactics  should  be 
dynamically  evolving  with  the  possible  choices  of  equipments.  How¬ 
ever,  this  is  not  the  domain  of  the  technical  side  of  the  Navy's  organ¬ 
isation.  Therefore,  it  is  necessary  to  coordinate  closely  with  opera¬ 
tional  divisions  to  keep  them  well  informed  of  technical  developments. 

Each  of  the  seven  variables  affects  the  information  flow  within 

the  submarine  and  is  therefore  important  in  the  determination  of  the 
required  functioned  information  flow  that  must  be  provided  in  the 
optimized  system  to  satisfy  the  threat  and  mission  requirements.  Once 
the  variables  to  be  considered  have  been  identified  in  the  tactical 
model,  the  laboratories  can  perform  a  state-of-the-art  survey  to 
determine  the  technological  capabilities  that  will  be  available  in  the 
given  time  period.  This  provides  limiting  parameters  to  the  tactical 
models  indicating  where  the  submarine's  performance  must  be  constrained 
by  the  state-of-the-art. 

The  next  step  is  to  embark  on  a  situation  analysis  of  each 
tactical  model  as  shown  in  Figure  4.  This  analysis  starts  with  the 
basic  hypotheses  of  the  model  and  then  treats  the  various  alternatives 
that  could  occur.  In  the  model  described  previously,  the  start  of  the 
evolution  occurs  when  contact  between  the  two  submarines  is  made.  For 
such  a  case  the  target  can  have  been  alerted  or  not.  If  alerted, 
it  can  attack  or  evade.  In  any  case,  own  ship  can  either  maintain 
or  lose  contact,  attack  or  track,  etc.  The  various  tie-ins  between 
the  alternatives  are  indicated  by  the  dashed  lines.  Mast  of  the  major 
alternatives  must  be  treated  if  the  system  design  is  to  be  optimized 
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for  a  maximum  number  of  situations. 


After  situation  analyses  have  been  performed,  two  types  of 
pictorials  are  brought  into  use,  operational  and  stress.  The  operational 
pictorial  (Figure  5)  is  a  picture  of  the  situation  first  described 
in  the  tactical  model  then  made  dynamic  in  the  situation  analysis. 

It  is  based  on  sound  operational  doctrine  derived  from  what  is  normally 
done  when  confronted  with  a  certain  situation.  This  is  useful  in 
observing  system  operation  under  normal  conditions.  However,  equip¬ 
ment  designers  are  concerned  with  stress  situations  in  which  various 
alternatives  can  be  considered  to  determine  vhat  particular  course  of 
action  produces  the  maximum  stress  in  technical  capabilities  of  the 
different  subsystems.  For  example,  firing  a. certain  weapon  at  its 
maximum  range  places  certain  requirements  on  the  performance  or 
accuracies  of  the  ship  control,  navigation,  and  sonar  subsystems. 
Similarly,  there  will  be  situations  which  place  extreme  demands  on 
ship  control  (avoidance  of  broaching),  navigation  (under  ice  navigating), 
and  each  of  the  other  subsystems  aboard  the  submarine.  These  stress 
situations  may  all  be  different  and  each  one  mu~t  be  studied.  These 
situations  are  displayed  by  stress  pictorials. 

At  this  point  functional  sequence  diagrams  (FSD’s,,  such  as 
shown  in  Figure  S,  are  produced.  These  diagrams  are  intended  to 
isolate  the  major  functions  which  are  performed  on  the  submarine, 
without  consideration  of  equipments  or  personnel,  and  trace  the  infor¬ 
mation  flow  among  them  so  that  it  is  possible  to  observe  how  the 
functions  are  coordinated  in  performing  a  given  job.  The  functional 
titles  on  the  FSD's  are  initially  drawn  st  a  gross  level,  e.g.,  weapon 


control,  surveillance,  countermeasures.  These  titles  correspond  to  the 
assignments  of  specialty  areas  to  the  Navy  laboratories.  Using  the 
FSD's,  the  personnel  in  a  given  laboratory  can  see  what  is  expected 
of  them  in  the  areas  assigned  to  their  responsibility  and  also  can  gain 
an  understanding  as  to  the  relationship  of  their  soecialty  to  those  of 
other  labs.  With  this  tool,  they  gain  an  understanding  of  how  informa¬ 
tion  generated  within  a  given  area  is  used  in  another  area. 

Upon  receiving  this  gross  first  level  FSD,  the  personnel  at  each 
laboratory  produce  a  second  level  FSD  (Figure  7)  to  identify  in  greater 
detail  the  relationships  between  subfunctions  contained  within  a  given 
function.  For  example,  the  ship  control  column  can  be  subdivided  into 
such  subfunctions  as  course  control,  speed  control,  depth  control,  and 
ballast  jontrol.  It  would  be  the  job  of  the  ship  control  laboratory 
to  relate  these  subfunctions  so  as  to  meet  the  functional  goals. 

Each  of  the  subfunctions  can  be  further  subdivided  into  lower 
levels  of  detail  as  illustrated  for  depth  control  in  Figure  8.  This 
subdividing  process  continues  until  the  lowest  meaningful  level  of 
subdivision,  called  the  n™1  level,  is  reached. 

At  the  lower  levels  of  detail,  it  becomes  possible  for  the  special¬ 
ists  at  the  laboratories  to  assign  specific  values  to  the  performance  or 
accuracies  of  the  elements  in  the  subsystem  based  on  the  tactical  situ¬ 
ation  under  study.  For  example,  in  a  given  situation  where  the  sub¬ 
marines  diving  planes  must  be  positioned  to  establish  a  trim  angle,  it 
may  be  necessary  to  establish  the  trim  angle  to  an  accuracy  of  +  0.5 
degrees.  For  another  situation  the  required  angle  might  be  quite  dif¬ 
ferent.  This  type  of  data  is  summed  in  the  "data  quality"  colunn  for 
all  postulated  situations,  to  see  what  the  actual  vaLLues  required  of  ship 
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system  parameters  are.  A  series  of  curves  are  derived  on  vhich  cost- 
effectiveness  studies  are  based  to  determine  the  accuracies  required 
for  the  overall  system.  It  may  be  possible  to  show  that  it  is  better 
to  avoid  a  situation  which  requires  extremely  stringent  accuracies  rather 
than  try  to  build  the  accuracy  capability  into  the  equipment.  Thus, 
these  data  quality  curves  form  the  basis  for  a  study  on  the  trade-offs 
possible  in  the  design  of  a  ship. 

Another  tool,  used  in  conjunction  with  the  functional  sequence 
diagrams, is  the  timing  sequence  diagram.  An  example  of  such  a  diagram 
is  illustrated  in  Figure  9.  The  FSD  does  not  show  the  amount  of  time 
that  it  takes  to  perform  a  given  operation  or  whether  certain  operations 
must  be  performed  continuously.  The  timing  sequence  diagram  shows  the 
typical  elapsed  time  to  perform  a  given  function,  therein  illustrating 
whether  the  timing  requirements  are  stressing  the  system. 

Once  the  functional  data  is  obtained  at  the  lowest  level,  it  is 
possible  to  bring  all  the  specialty  areas  together  to  observe  the  detailed 
information  flow  for  a  given  operation  as  shown  in  Figure  10.  It  is  at 
this  point  that  accidental  functional  duplication  will  b  apparent  as 
two  or  more  subsystems  ere  seen  to  be  performing  the  same  function.  Data 
quality  mismatches  will  also  become  obvious  if  one  subsystem  requires  data 
to  a  given  degree  of  accuracy  but  it  is  being  supplied  with  a  different 
accuracy  from  another  subsystem.  It  is  possible  that  one  subsystem  may 
be  found  not  to  be  providing  data  to  another  subsystem  requiring  it  since 
this  requirement  was  not  previously  known. 

After  this  stage  of  design  has  been  completed,  ail  the  nth  level  FSD* 

are  sunmed  over  all  the  different  situations  to  end  up  with  one  functional 
system  block  diagram.  The  data  for  the  construction  of  the  block  diagram 
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is  determined  from  the  FSD's.  Now  it  is  possible  to  isolate  another, 
more  subtile,  duplication  of  functions  which  occurs  where  a  mathematical 
process  being  performed  in  one  subsystem  is  analogous  to  that  done  in 
another  subsystem. 

After  the  system  block  diagram  is  completed,  a  traffic  flow 
analysis  is  performed  to  reveal  how  often  the  various  functions  are 
used  together  in  different  situations.  Thus  a  count  is  obtained  on 
the  number  of  times  function  A  is  related  to  function  B  or  function 
B  to  function  C,  etc.  This  analysis  may  show,  for  example,  that  certain 
functions  within  the  ship  control  subsystem  relate  to  functions  in 
weapon  control  much  more  often  than  they  do  to  other  functions  within 
ship  control.  Therefore,  those  functions  should  be  regrouped  with  the 
weapon  control  subsystem.  Thus,  functions  that  are  frequently  used 
together  can  be  grouped  together  to  form  optional  subsystems.  In 
this  way,  subsystems  are  determined  on  a  functional  information  flow 
basis  rather  than  on  a  historical,  and  possibly  outdated,  one. 

Once  the  functional  subsystems  have  been  defined,  the  subsysten 
and  system  syntheses  are  performed.  Various  configurations  of  men 
and  machines  are  postulated,  trade-off  and  cost-effectiveness  analyses 
performed,  and  the  final  optimized  system  determined.  At  this  point, 
equipnent  and  personnel  are  specifically  named  to  perform  all  the  re¬ 
quired  functions.  Given  this  data,  the  operational  sequence  diagram 
(OSD),  such  as  shown  in  Figure  11,  may  now  be  employed.  With  this 
tool  which  was  originally  developed  by  Dunlop  Associates  in  1 959,  the 

information  flow  is  again  traced  but  now  the  column  headings  represent 
"equipments"  and  "people"  rather  than  "functions."  Symbology  is  used 

to  describe  the  manner  in  which  information  is  transferred  —  electrically. 


visually,  audibly,  tactually,  etc.  Any  equipment  mismatches,  previously 
overlooked,  will  become  apparent  at  this  point.  One  can  observe  the 
column  on  the  diagram  corresponding  to  any  person  aboard  the  submarine 
to  determine  whether  he  is  overloaded  in  performing  a  certain  operation. 
Perhaps,  another  may  be  needed  to  share  his  duties,  or  if  he  is  under¬ 
loaded  he  might  share  the  duties  of  someone  else  who  is  overloaded  to 
reduce  manning  requirements  aboard  the  ship. 

Upon  completion  of  the  OSD's,  the  next  step  is  to  use  multiple 
correlation  charting.  This  chart,  which  is  believed  to  have  been 
originated  by  the  British,  is  used  to  determine  the  specific  arrange¬ 
ment  of  men  and  equipment  aboard  the  submarine.  Vertical  and  hori¬ 
zontal  chart  headings  are  identical  and  represent  the  men  and  equipments 
studied  in  the  operational  sequence  diagrams.  Through  use  of  this  tool, 
it  is  possible  to  provide  adjacencies  (and  relationships)  as  required. 

Figure  12  shows  the  path  by  which  an  operational  sequence  diagram 
is  used  to  form  a  correlation  chart  which  in  turn  yields  a  layout  with 
optimized  information  flow. 

The  sun  of  all  the  previously  mentioned  analytical  techniques 
makes  up  the  FRISCO  system  design  procedure.  It  is  important  to 
note  that  none  of  these  techniques  automatically  perform  the  engineering 
of  the  system.  The  same  talented  engineers  are  still  required  in  the 
design  process.  Bbwever,  the  techniques  provide,  at  the  very  least,  a 
communications  tool  for  people  who  are  working  towards  the  same  goal 
but  are  geographically  separated  by  large  distances.  They  also  provide 
a  method  by  which  interface  problems  can  be  isolated  and  resolved. 

It  is  believed  that  these  techniques  should  prove  to  be  very  useful 
in  systems  design.  There  is  as  yet  no  evidence  of  this  in  FRISCO  since 
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RELIABILITY  AND  MAINTAINABILITY  CONSIDERATIONS  IN  SYSTEMS  DESIGN 


The  need  for  a  systems  engineering  approach  in  the  design  of  Naval  War¬ 
fare  Systems  is  becoming  increasingly  apparent.  We  in  the  Naval  Material 
Support  Establishment  are  being  asked  to  make  trade-offs  of  basic  system 
effectiveness  parameters  like  Performance,  Reliability  and  Maintainability t 
These  trade-offs  are  made  with  respect  to  budget,  manpower,  and  physical  con¬ 
straints  where  cost  includes  operational  as  well  as  investment  dollars  and 
manpower  considerations  include  skill  level  as  well  as  numbers  of  technical 
people.  (Slide  1) 

The  optimization  problem  is  further  complicated  by  the  time  parameter  in 
that  system  development  is  an  evolutionary  process  where  the  parameters  to  be 
considered  and  the  data  available  to  guide  decisions  will  vary  in  detail  and 
significance  as  a  function  of  the  particular  point  in  the  development  cycle  one 
may  be  at.  Also,  with  respect  to  time  as  a  parameter,  the  operational  need  for 
a  system  will  impose  a  lead  time  constraint  which  in  some  cases  could  become 
the  overriding  constraint. 

Finally,  for  trade-offs  to  be  meaningful,  all  parameters  must  be  related 
to  specific  measures  of  effectiveness.  Detailed  studies  underway  at  the  Naval 
Applied  Science  Laboratory  for  the  ‘PACED  program  are  aimed  at  the  development 
of  analytic  techniques  for  guiding  the  program- manager  in  making  these  neces¬ 
sary  trade-offs.  It  has  been  found  that  the  major  problem  to  be  solved  is 
that  of  defining  the  specific  mission  oriented  or  operational  measures  of 
effectiveness  and  relating  these  measures  to  parameters  which  the  design 
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engineer  can  work  to.  This  translation  of  effectiveness  measures  will  help 
solve  the  conmunication  problem  which  exists  between  the  producers  and  the 
users  of  Warfare  Systems. 


Looking  at  one  of  the  parameters  of  systems  effectiveness,,  namely, 
operational  reliability, one  can  readily  see  how  failure  to  relate  .parameter 
values  to  mission  tasks  can  lead  to  erroneous  trade-offs.  (Slide  2)  The 
measure  availability, for  example,  implies  that  repair  rate  and  failure  rate 
reductions  can  equally  improve  the  probability  of  having  a  system  operational 
at  a  point  in  time.  However,  if  the  mission  requires  sustained  performance  for 
a  particular  duration  then  the  product  of  availability  and  reliability  becomes 
the  measure  of  success  and  the  failure  rate  becomes  the  more  critical  parameter. 
Finally,  if  mission  analysis  shows  -hat  certain  maximum  maintenance  time  con¬ 
straints  are  allowable  without  mission  degradation,  then  repair  rate  again  be¬ 
comes  critical. 


The  PACED  program  is  a  broad  program  which  is  looking  at  all  the  facets 
of  system  effectiveness.  (Slide  3)  Briefly,  we  are  looking  at  system  develop¬ 
ment  life  cycles  from  concept  to  operational  phases.  Various  studies  in  exist- 
ance  are  attempting  to  uniquely  define  a  formalized  development  cycle.  PACED 
has  identified  four  major  points  in  this  cycle  occurring  during  the  concept 
and  development  phases  in  order  to  study  the  time  related  problems  of  system 
development.  Engineering  studies  are  underway  to  develop  Design  Disclosure 
Formats  for  complete  disclosure  of  the  s>?tem  at  these  points  in  order  that  a 
proper  information  source  could  be  available  to  facilitate  utilization  of 
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analytic  techniques  for  major  trade-off  decisions.  Data  accumulated  through 
studies  of  automatic  test  concepts,  microelectronics,  packaging,  and  high  power 
devices  will  be  made  available  to  program  managers  for  introduction  to  new 
systems  at  the  appropriate  point  in  the  development  process. 

The  purpose  of  this  paper  is  to  describe  those  portions  of  the  PACED 
program  specifically  oriented  toward  providing  concepts  and  techniques  which 
would  enable  the  introduction  of  Reliability  and  Maintainability  into  system 
design. 


The  first  important  consideration,  of  course,  is  the  development  of 
requirements  for  reliability  and  maintainability.  The  problem  here  goes  beyond 
merely  stating  goals  for  our  system.  Reliability  and  Maintainability,  when 
specified,  must  be  considered  contractually  in  the  same  manner  as  any  other 
system  parameter.  This  implies  that  attitudes  must  be  properly  orientedt  which 
further  implies  that  reliability  training  must  be  actively  pursued. 

The  second  important  consideration  involved  in  successfully  introducing 
Reliability  and  Maintainability  to  design  lies  in  active  participation  during 
the  development  process.  Today's  systems  have  become  so  complex  as  to  require 
much  more  than  specifications  and  acceptance  testing  by  the  Military.  We  must 
become  technical  managers  of  our  systems  and  through  carefully  planned  design 
reviews  we  must  actively  participate  during  the  evolutionary  process  of  design. 
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To  participate  we  need: 

1.  A  store  o£  data  and  knowledges  to  guide  our  decisions  and 

2.  A  method  of  communication. 

The  data  we  need  must  consist  of  information  in  the  areas  where  possible  solu¬ 
tions  to  the  Reliability/Maintainability  problem  may  be  found.  These  solutions 
take  many  forms.  (Slide  4)  The  degree  and  type  of  automatic  testing,  the 
mechanization  techniques,  the  kind  of  redundancy,  the  use  of  de-rating,  and 
the  use  of  microelectronics  are  some  of  the  possible  solutions  available  to  the 
program  manager.  If  the  designer  is  to  take  advantage  of  any  proposed  scheme 
he  must  have  quantitative  data  which  describes  the  approach  in  terms  of  his 
basic  measures  of  effectiveness.  For  example,  the  impact  of  microelectronics 
and  modular  design  must  be  known  not  only  in  terms  of  failure  rates  and  ease  of 
maintenance  but  also  in  terms  of  cost  and  effect  on  logistics.  Alternate  test 
concepts  must  be  disclosed  in  terms  of  effect  on  space,  people  and  budget  as 
well  as  prime  system  repair  times. 

An  example  of  how  one  could  assist  the  design  engineer  through  disclosure 
of  state  of  the  art  information  in  a  useful  form  is  shown  in  the  PACED  efforts 
related  to  microelectronics  and  modular  design  (slide  5).  In  order  to  support 
the  program  manager  in  this  area,  current  and  planned  packaging  concepts  are 
being  described  and  documented  in  a  Design  Disclosure  Workbook.  Care  is  being 
given  to  detailing  basic  parameters  so  that  alternate  systems  could  be  synthe¬ 
sized  from  this  information.  Specific  knowledges  gained  by  engineers  who 
assisted  in  the  design  of  these  systems  is  also  being  documented  through  regu¬ 
lar  working  meetings  of  a  packaging  committee  set  up  by  PACED  with  membership 
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from  Bureau  of  Ships*  Bureau  of  Naval  Weapons*  Office  of  Naval  Research  and 
their  Laboratories.  This  Workbook  will  be  supplemented  by  a  library  of  avail¬ 
able  circuit  functions  of  modular  designs*  available  microelectronic  circuits* 
and  functional  makeup  of  major  Naval  Systems.  Correlation  programs  have  been 
developed  so  that  data  processing  equipment  could  be  used  for  correlating  pro¬ 
posed  functional  boundaries  for  new  designs  against  available  designs  and 
available  microelectronic  circuits.  The  microelectronic  functions  encoded 
include  data  from  National  Aeronautics  and  Space  Administration*  Air  Force  and 
Army  programs.  Although  the  long  range  objectives  of  this  portion  of  the  PACED 
program  are  to  develop  specific  mechanization  techniques*  it  is  important  to 
note  that  the  early  availability  of  data, via  the  Design  Disclosure  Workbook 
for  packaging  parameters,  and,, the  Data  Library  for  circuit  functions,*  will  enable 
Naval  system  designers  to  introduce  available  concepts  to  current  designs.  In 
order  to  be  of  general  use  to  Laboratories  and  contractors  the  library  will  be 
maintained  by  the  University  qf  Pennsylvania.  A  handbook  on  its  applicability 
and  use  will  be  published  ea^ly  in  196S. 

The  second  requirement  mentioned  for  allowing  active  effective  participa¬ 
tion  in  the  development  process  is  the  need  for  a  means  of  conmunication  between 
the  designers  of  our  systems  and  the  technical  managers  of  our  programs.  This 
need  became  very  apparent  to  the  Laboratory  during  the  planning  stages  of  a 
total, support  system  for  advanced  ASW  surface  ships  and  submarines.  We  learned 
that  support  systems  and  support  system  concepts  could  not  be  applied  without 
sufficient  knowledge  of  the  prime  system.  In  addition,  in  order  to  avoid 
costly  retrofits  the  information  must  be  available  during  the  early  design 


phases.  The  maintainability  tasks  developed  by  tne  PACED  program  recognized 
this  need  and  emphasized  in  support  system  development  the  requirement  for 
documentation  which  we  call  Design  Disclosure  Format  (DDF).  The  philosophy 
adopted  is  straightforward,  namely i  Once  a  requirement  for  maintainability 
has  been  specified,  then  the  responsibility  for  achieving  this  is  clearly  with 
the  prime  developer.  The  technical  managers  of  the  program  must  be  able  to 
then  review  the  proposed  design  for  maintainability  and  introduce  specific  hard¬ 
ware  concepts  where  needed  to  effect  necessary  improvements.  The  contractor 
documentation  must  allow  rapid  assessment  of  the  maintainability  of  his  design 
as  well  as  indicate  the  need  for  support  equipment  and  support  personnel.  (Slide  6) 

The  DDF  requirements  have  been  generated  to  present  only  necessary  infor¬ 
mation.  Through  elimination  of  unwanted  data  it  is  expected  that  overall  con¬ 
tractor  documentation  requirements  will  be  actually  reduced  in  terms  of  volume 
and  cost.  The  DDF  requires  presentation  of  functional  circuits  and  physical 
boundaries  to  allow  hardware  interface  analyses.  Operational,  test,  and  power 
circuits  are  independently  identified  and  charts  showing  operational  sequences 
and  circuit  dependency  are  required.  (Slide  7) 

The  operational  sequence  diagrams  are  currently  being  developed  by  the 
Bureau  of  Ships  Work  Study  Program  to  enable  their  use  as  a  design  tool.  This 
work  is  being  done  jointly  with  the  FRISCO  program. 


The  dependency  chart  being  developed  by  the  PACED  program  provides  the 
tool  for  maintainability  analysis  during  design.  This  information  will  allow 
system  analysis  of  the  maintenance  tasks  for  an  entire  shipboard  suit  using  a 
uniform  format  and  will  result  in  better  integration  of  the  maintenance  function 
(utilization  of  inherent  sensors,  computer  capacity,  etc.).  In  particular,  the 
availability  of  a  Circuit  and  Indicator  Dependency  Chart  early  in  the  acquisition 
cycle  before  designs  become  frozen  would  have  a  major  impact  on  improvement  of 
the  ultimate  design  of  equipment.  Design  adequacy  with  respect  to  test  point 
selection  and  the  ease  with  which  a  fault  can  be  bracketed  to  simplify  potential 
man-machine  interfaces  will  be  enhanced,  so  that  problems  can  be  quickly  dis¬ 
cerned  and  effective  design  modifications  made.  (Slide  8) 

A  preliminary  report  on  this  technique  has  been  issued  and  a  detailed 
technical  report  describing  DDF  content  and  intended  application  is  currently 
being  reviewed  for  release  in  January  1965.  Coordination  has  been  started  with 
the  Bureau  of  Naval  Personnel  and  their  field  activities  and  program  managers 
within  the  Bureau  of  Naval  Weapons  (A-NEW  Program),  the  Bureau  of  Ships  (SEAHAWK, 
FRISCO,  SOUTHERN  CROSS  Programs)  and  Defense  Communications  Agency  (DCA) (COMSAT 
Program)  have  been  technically  oriented  on  DDF  use  for  support  system  develop¬ 
ment. 

In  summary,  PACED  is  specifically  addressing  itself  to  how  one  introduces 
Reliability  and  Maintainability  to  system  design.  The  solution  lies  in  taking 
the  simple  statement  "apply  state  of  the  art  through  effective  liaison"  and 
making  it  work.  This  involves  the  development  of  techniques  described  herein 
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for  documenting  information  and  allowing  effective  communication.  These 
techniques  have  been  presented  to  Industry  through  EIA  conferences  and  private 
meetings.  The  implementation  of  these  techniques  will  be  validated  on  selected 
subsystems.  It  is  hoped  that  through  application  over  time  the  techniques  will 
be  improved  and  ultimately  contribute  to  the  solution  of  current  problems  in 
the  application  of  Reliability  and  Maintainability  to  system  design. 
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ENGINEERING  INTEGRATION  IN  SYSTEM  DESIGN 


Introduction:  You  have  heard  previous  discussions  this  afternoon 
concerning  some  facets  of  System  Effectiveness,  namely  reliability 
and  maintainability,  Also  discussed  was  the  use  of  analytical  techniques 
in  determining  the  optimum  system  design  to  insure  mission  success.  Here, 
the  threat  is  analyzed  and  then  various  system  configurations  are 
evaluated  through  the  use  of  threat  and  performance  models.  To  bring 
the  concept  of  System  Effectiveness  more  into  focus,  I  would  like  to 
summarize  what  I  consider  the  pertinent  elements  of  System  Effectiveness. 

(Slide  #1  “Elements  of  System  Effectiveness”) 

Elements  of  System  Effectiveness:  Simply  stated  System  Effectiveness 
is  defined  as  ”the  probability  that  the  system  (or  material)  will  operate 
successfully  when  required  under  specific  conditions."  I  am  using  the 
term  system  in  the  gross  sense.  The  primary  contributors  to  Effectiveness 
are  PERFORMANCE,  AVAILABILITY  and  UTILIZATION.  PERFORMANCE  indicates 
"How  Well"  the  system  operates.  AVAILABILITY  indicates  "How  Long"  the 
system  can  function  under  certain  conditions.  UTILIZATION  indicates 
"How  Often"  the  system  will  be  used.  Other  supporting  factors  associated 
with  the  primary  elements  include  such  items  as  design  adequacy,  man- 
machine  interfaces,  equipment  reliability,  serviceability,  and  environment. 
Of  the  three  primary  elements  of  System  Effectiveness,  the  PERFORMANCE 
factor  is  the  one  that  I  would  like  to  emphasize  today. 

Performance  Through  Design:  In  order  to  achieve  the  goal  of  PERFORMANCE, 
one  must  first  consider  system  design,  since  design  is  the  conjugate  of 
PERFORMANCE.  In  addition  to  considering  the  design  of  the  device  or 
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subsystem  under  development,  the  parameters  of  associated  peripheral 
equipment  or  subsystems  must  be  analyzed.  I  am  using  the  terra  subsystem 
to  emphasize  that  any  system  is  a  part  of  a  bigger  system.  A  systematic 
engineering  approach  is  required  to  insure  complete  interface  compati¬ 
bility,  between  major  equipments.  This  engineering  approach  is  called 
Engineering  Integration.  The  key  to  this  design  appears  to  be  through 
a  System  Effectiveness  analysis  of  the  total  systems  of  which  new 
developments  would  comprise  a  part.  This  analysis  can  help  to  focus 
attention  on  interface  requirements  in  terms  of  performance  and  compati¬ 
bility.  For  example,  starting  with  the  operational  requirement  of  a 
weapons  system,  one  can  analyze  the  operational  requirements  of  associated 
subsystems  to  determine  such  factors  as  overlapping  requirements, 
redundancy,  deficiencies,  mutual  support,  incompatibility  and  the  like. 
Additional  questions  such  as  the  following  might  be  asked.  In  terms  of 
UTILIZATION,  has  subsystem  A  been  designed  for  the  same  mission  length 
as  subsystem  B  (which  is  complementary  and/or  interdependent)  for  carrying 
out  a  specific  mission?  Are  the  same  shipboard  subsystems  being  designed 
for  the  same  environment?  Will  complementary  subsystems  possess  similar 
availability  or  operational  readiness  factors?  Does  subsystem  A  have  a 
material  reliability  of  50$  while  System  B,  which  is  dependent  oh  sub¬ 
system  A,  have  a  reliability  of  95$  and  the  overall  mission  reliability 
required  is  90$?  Further,  in  regards  to  PERFORMANCE,  are  connecting  sub¬ 
systems  compatible  or  is  subsystem  A  providing  a  digital  output  while 
existing  subsystem  B  was  designed  previously  to  accept  only  an  analog  input? 


Are  man-machine  interfaces  being  considered  initially  in  the  design  or 
may  operation  of  the  equipments  being  developed  result  in  undo  strain 
and  operator  fatigue  which  uill  result  in  decreased  Effectiveness? 

As  you  see  from  the  foregoing  all  aspects  of  System  Effectiveness 
can  and  must  be  considered.  PERFORMANCE  must  compete  with  AVAILABILITY 
and  UTILIZATION  and  vice  versa,  but  with  each  given  its  due  weight.  The 
Navy  can  no  longer  emphasize  one  segment  of  effectiveness  at  the  expense 
of  the  other  without  regard  to  their  relative  contribution  to  mission 
completion. 

Design  Approach;  Today  I  would  like  to  briefly  suggest  some  methods 
of  approach  to  insure:  (l)  engineering  integration  of  subsystems  making 
up  a  whole  system;  and  (2)  compatible  mating  of  the  subsystem  with  the 
engineering  interfaces  connecting  it  to  other  mutually  dependent  sub¬ 
systems.  These  techniques  could  be  equally  applied  to  the  AVAILABILITY 
and  UTILIZATION  aspects.  The  ideal  method  to  approach  the  problem 
would  be  to  consider  the  complete  system  (Slide  #2).  Here,  all  aspects 
of  design  can  be  treated  simultaneously.  Unfortunately,  since  major 
portions  of  tre  system  already  exist  the  new  subsystem  must  be  injected 
in  a  piecemeal  fashion.  Nevertheless  to  minimize  redes:'  and  patching, 
a  time-sequenced  approach  can  be  utilized  (Slide  #3).  ■  system 

design  is  considered,  however,  but  the  implementation  is  Lime-phased  as 
resources  permit.  As  future  system  changes  evolve  due  to  changing  needs, 
they  can  be  assessed  prior  to  their  introduction  in  the  system  performance 
model.  A  critical  examination  of  the  change  prior  to -implementation  can 
insure  minimum  physical  and  functional  disruption,  provide  smooth 


Are  man-machine  interfaces  being  considered  initially  in  the  design  or 
may  operation  of  the  equipments  being  developed  result  in  undo  strain 
and  operator  fatigue  which  will  result  in  decreased  Effectiveness? 

As  you  see  from  the  foregoing  all  aspects  of  System  Effectiveness 
can  and  must  be  considered.  PERFORMANCE  must  compete  with  AVAILABILITY 
and  UTILIZATION  and  vice  versa,  but  with  each  given  its  due  weight.  The 
Navy  can  no  longer  emphasize  one  segment  of  effectiveness  at  the  expense 
of  the  other  without  regard  to  their  relative  contribution  to  mission 
completion. 

Design  Approach:  Today  I  would  like  to  briefly  suggest  some  methods 
of  approach  to  insure:  (l)  engineering  integration  of  subsystems  making 
up  a  whole  system;  and  (2)  compatible  mating  of  the  subsystem  with  the 
engineering  interfaces  connecting  it  to  other  mutually  dependent  sub¬ 
systems.  These  techniques  could  be  equally  applied  to  the  AVAILABILITY 
and  UTILIZATION  aspects.  The  ideal  method  to  approach  the  problem 
would  be  to  consider  the  complete  system  (Slide  #2).  Here,  all  aspects 
of  design  can  be  treated  simultaneously.  Unfortunately,  since  major 
portions  of  tre  system  already  exist  the  new  subsystem  must  be  injected 
in  a  piecemeal  fashion.  Nevertheless  t-';  minimize  redes:'  and  patching, 
a  time-sequenced  approach  can  be  utilized  (Slide  #3).  .  system 
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resources  permit.  As  future  system  changes  evolve  due  to  changing  needs, 
they  can  be  assessed  prior  to  their  introduction  in  the  system  performance 
model.  A  critical  examination  of  the  change  prior  to -implementation  can 
insure  minimum  physical  and  functional  disruption,  provide  smooth 


TIME  PHASED  SYSTEM  DESIGN 


Di  3 


(NEW)  (EXISTING)  (EXISTING) 


integration  and  minimize  premature  commitment  of  marginal  changes. 
Functional  Block  Diagram:  The  ability  of  a  system  to  accomplish  its 
designated  functions  is  described  as  PERFORMANCE  and  can  be  measured  by 
the  assessment  of  performance  models.  The  subsystem  performance  require¬ 
ments  must  first  be  translated  into  system  design  requirements  and  from 
there  into  design  specifications.  Generally  an  analysis  of  the  functions 
of  a  system  leads  to  the  development  of  a  system  functional  block 
diagram.  For  the  first  cut,  only  the  major  functions  are  considered, 
(Slide  #4).  The  major  blocks  are  in  turn  further  broken  up  in  successive 
steps  as  required  to  identify  signal  flow,  transfer  functions  and 
tolerances. 

The  performance  factors  should  be  listed  by  each  block  along  with 
the  design  parameters  and  applicable  mathematical  relationships,  (Slide 
#5).  Once  the  subsystem  under  development  has  been  carefully  analyzed, 
the  next  step  is  to  go  through  a  similar  analysis  wth  connecting  sub¬ 
systems  via  the  interfaces.  Here  again  performance  requirements  must 
be  analyzed  to  insure  that  this  new  development  will  enhance  overall 
performance  and  will  not  restrict  it.  Let  us  lake  the  example  of  a 
surveillance  radar  system.  One  area  to  be  analyzed  is  the  capability  of 
the  radar  to  detect  and  identify  the  target  in  sufficient  time  at  a  high 
enough  data  rate  so  that:  (l)  the  weapon  designation  equipment  can 
assess  the  target  threat  and  assign  it  to  a  missile  system;  and  (2)  allow 
the  missile  system  sufficient  time  to  react.  Once  it  has  been  determined 
that  the  performance  factors  will  improve  overall  system  performance, 
then  the  physical  and  functional  interfaces  must  be  made  compatible  with 
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each  other.  Tpe  process  of  using  functional  block  diagrams  is  repeated. 
Summarization  Diagrams 

While  the  mathematical  relationships  among  the  variables  can  be 
identified  by  using  system  block  diagrams,  other  interface  considerations 
such  as  reliability,  environmental  specifications  and  man-machine 
relations,  must  be  summarized.  This  may  be  done  in  form  of  PERT  type 
network  diagrams,  matrices,  or  sequential  diagrams. 

In  the  network  diagrams  (Slide  #6)  the  events  A,  B,  G,  etc., 
represent  variables  at  various  subsystem  levels.  The  activities 
denoted  by  1,  2,  3,  4,  etc.,  represent  the  functional  relationships 
which  exists  between  two  variables. 

The  same  type  of  information  can  be  expressed  in  a  matrix  form 
(Slide  #7).  Using  the  same  coding  as  the  network  diagram,  the  data  is 
presented  in  a  form  which  is  suitable  for  programming  into  a  computer. 

The  network  lends  itself  more  easily  to  visually  identifying  relation¬ 
ships  while  the  matrix  is  more  suitable  for  storage  and  retrieval. 

In  the  sequential  diagram  (Slide  #8)  relationships  based  on  factors 
such  as  processes,  information  sources,  decision  and  outcomes  can  be 
laid  out  in  step  fashion.  This  is  an  operational  analysis  type  of 
approach  which  pictorially  displays  information-decision-action-flow 
sequences  which  a  component  or  subsystem  must  undergo  to  complete  its 
mission.  This  type  diagram  is  now  commonly  used  in  man-machine  interface 
analysis . 

Other  methods  of  summarizing  both  subsystem  and  system  PERFORMANCE  - 
DESIGN  information  include  matrices  showing  PERFORMANCE  vs  DESIGN, 
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Since  subsystem  developments  are  not  always  under  the  management  of 
the  same  agency,  it  is  necessary  to  formally  ' ransmit  design  criteria 
from  one  agency  to  another.  This  can  be  done  by  using  Design  Interface 
Specifications  (.Slide  #10).  It  is  essential  that  this  be  done  as  soon 
as  possible  in  the  development  in  order  to  allow  the  receiving  agency 
sufficient  time  to  respond.  It  should  be  noted  that  there  are  always 
funding  implications  that  must  be  reconciled.  Interface  engineering 
can  be  an  expensive  proposition  and  should  not  be  overlooked.  The 
slide  shows  some  of  the  criteria  that  should  be  included  in  the  interface 
specification. 

Summary 

My  intent  today  has  been  to  give  you  some  feel  of  the  magnitude  of 
engineering  integration.  The  developer  of  a  subsystem  can  no  longer 
live  in  a  vacuum.  He  must  be  jointly  responsible  for  interface  design. 
Additionally,  the  Navy  must  apply  total  system  engineering  and  not 
piecemeal  engineering  of  individual  subsystems. 
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A  little  less  than  two  years  ago  I  participated  in  the  briefing 
of  a  Fleet  Commander  on  the  status  of  the  development  and  testing 
programs  of  various  weapons  and  support  systems.  At  the  conclusion 
of  the  briefing,  Admiral  H.  P.  Smith,  Commander-in-Chief  of  the 
Atlantic  Fleet,  made  a  few  closing  remarks  which  I  think  bear 
repeating. 

The  gist  of  the  Admirals'  remarks  was  as  follows: 

My  ships  are  burdened  with  so-called  sophisticated  equipment 
and  systems  which  have  wonderful  "press  clippings'*  concerning 
their  performance.  But  unfortunately,  they  won't  work  when  we 
need  them.  These  complex  systems  are  generally  unreliable  and 
very  difficult  to  maintain.  When  they  work  their  performance  is 
usually  quite  good.  However,  I  would  gladly  sacrifice  some 
performance  for  the  sake  of  reliability  and  maintainability.  My 
ships  need  equipment  and  systems  that  work  when  and  where  they 
are  needed  to  work.  They  don't  need  any  more  junk  installed  in 
them. 

Perhaps  out  of  sheer  frustration  Admiral  Smith  may  have  over¬ 
stated  the  situation.  Nevertheless,  there  is  more  than  a  little 
substance  to  his  comments.  The  situation  the  Admiral  describes 
exists  in  varying  degrees  in  the  Fleet  today.  What  it  amounts  to  is 
that  in  this  era  of  a  technological  boom  many  of  our  systems  were 
designed  and  developed  to  state-of-the-art  performance  limitations, 
rather  than  including  all  of  the  other  qualitative  elements  which  are 
characteristic  of  an  operationally  effective  system.  An  effective 
system  as  previously  defined  this  afternoon,  is  a  system  that  can 


successfully  meet  an  operational  demand  throughout  a  given  time 
period  when  operated  under  specified  conditions.  The  basic  qualita¬ 
tive  elements  which  contribute  to  this  systems  effectiveness,  as 
Mr.  Rohe  pointed  out  in  the  last  presentation,  includes  in  addition 
to  performance,  -  reliability,  maintainability,  compatability,  opera¬ 
bility,  logistics  supportability,  human  factors,  and  others.  I 
would  be  wrong  tc  state  that  these  elements  were  not  considered 
at  sometime  and  in  seme  way  during  the  course  of  development  or 
production  of  *  'day's  systems.  However,  I  will  state  that  the  inclusion 
of  these  elements  into  system  development  was  not  adequately  planned 
for,  resulting  in  an  ill-timed  and  fragmented  approach  to  systems 
effectiveness.  I  believe  this  statement  is  certainly  justified  in 
view  of  the  number  of  "get  well"  programs  which  are  now  underway 
with  many  of  our  major  systems.  Basically,  the  purpose  of  these 
"get  well"  programs  is  to  retrofit  effectiveness  into  the  systems, 
however,  "get  well"  programs  actually  accomplish  little  more  than 
the  incineration  of  funds  which  we  could  be  using  more  productively 
elsewhere  in  our  RDT&E  programs.  You  cannot  drill  a  hole  in  a  black 
box  and  stuff  in  some  reliability  or  add  another  black  box  and  label 
it  maintainability  or  compatibility,  or  pasteon  some  operability. 

What  then  is  the  solution  to  this  problem  of  being  able  to 
provide  operational  ly  effective  systems  to  the  Fleet?  Systems 
that  will  not  only  meet  performance  specifications  but  will  operate 
when  called  upon  to  operate  without  the  necessity  of  having 
contractor  technicians,  engineers,  or  PhD's  available  to  keep  them 
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goi^g.  There  must  be  a  solution.  We  may  never  find  an  infallible 
solution  or  a  panacea  to  this  situation  —  but  we  must  strive  for 
one. 

A  solution  to  the  problem  cannot  be  achieved  without  the 
integration  in  design  and  development  planning  of  the  disciplines  — 
if  you  will  —  which  constitute  system  effectiveness  such  as  reliabi¬ 
lity,  maintainability,  etc.  W r  must  break  with  the  tradition  of 
treating  these  as  separ. functional  entities.  In  addition  this 
integration  process  must  be  instituted  at  the  conception  of  a 
system  —  i.e.  we  must  start  early;  at  the  beginning  —  and  the 
resulting  plans  refined  throughout  the  entire  development  planning 
phase  of  the  system.  I  submit  that  this  integration  early  in  the 
gestation  period  of  a  system  should  result  in  viable  trade-offs 
between  these  disciplines  and  pure  performance  objectives.  This 
permits  the  optimization  of  true  operational  effectiveness.  However, 
I  must  qualify  this  submission  slightly.  I  have  to  define  what  I 
mean  by  the  "system"  that  I  refer  to.  The  system  considered  must 
necessarily  be  the  total  system.  It  must  not  only  be  comprised 
of  the  hardware  blai..k  boxes  we  are  going  to  eventually  desi.cn  ar.d 
assemble  but  it  must  take  into  consideration  the  black  boxes  of 
other  systems  or  sub-systems  which  may  supply  support  or  be 
supported.  The  eventual  overall  operational  configuration  must 
considered  which  includes,  in  addition  to  equipment  interfaces, 
such  things  as  physical  location,  environraent;both  functional  and 
pnysical,  and  mission  requirements. 
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It  is  extremely  difficult  and  nigh  on  to  impossible  to  scribe 
a  circle  and  state  this  is  the  total  system,  so  permit  me  to  take 
a  simple-minded  out  by  calling  it  the  big  picture  or  the  integrated 
whole. 

The  Department  of  Defense  and  the  Department  of  the  Navy 
recently  have  taken  steps  to  incorporate  regulations  and  procedures 
in  the  RDT&E  plannir^g  process  to  provide  for  this  integrated,  total 
system  approach  in  the  development  of  Naval  Weapons  and  support 
systems. 

In  February  of  this  year  the  Department  of  Defense  promulgated 
a  new  directive  which  directs  that  all  new  or  major  modifications 
of  existing  Engineering  Development  or  Operational  System  Develop¬ 
ment  projects  estimated  to  require  cumulative  RDT&E  financing  in 
excess  of  twenty-five  million  dollars,  or  estimated  to  require 
production  investment  in  excess  of  one  hundred  million  dollars 
shall  include  a  Project  Definition  Phase  or  PDP  as  it  is  commonly 
abbreviated.  This  requirement  may  be  specifically  waived  by 
written  approval  of  the  Director  of  Defense  for  Research  and  En¬ 
gineering.  Other  projects  may  be  required  to  include  a  PDP,  in 
whole  or  in  part,  at  the  discretion  of  the  Department  of  the  Navy  or 
as  directed  by  the  Department  of  Defense  for  Research  and  Engineering 
(DDR&E) .  The  PDP  is  one  of  the  formal  steps  in  the  development 
planning  process  during  which  preliminary  engineering,  and  contract 
and  management  planning  are  accomplished  in  an  environment  that 
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encourages  realism  and  objectivity. 

The  objectives  of  a  PDP  are  to  establish  trade-offs  within 
the  mission  and  performance  envelopes;  establish  firm  and  realistic 
specifications;  precisely  define  interfaces  and  responsibilities; 
identify  high-risk  areas;  select  the  best  technical  approaches; 
establish  firm  and  realistic  schedules  and  cost  estimates;  and 
achieve  fixed-price  or  incentive  contracts  for  the  subsequent 
full-scale  development  phase  of  the  project.  The  results  of  a 
PDP  must  provide  an  adequate  basis  to  ensure  that  management  decisions 
to  proceed  with  or  cancel  or  change  projects  are  made  on  the  basis 
of  a  total  system  and  total  cost  basis,  realistic  schedule  estimates 
and  achievable  performance  specifications. 

PDP  is  at  least  a  partial  solution  to  the  poor  planning,  un¬ 
realistic  schedules,  unanticipated  design  changes,  large  increase 
in  costs  over  original  estimates  and  "get  well"  programs  which  un¬ 
fortunately  have  been  characteristic  of  the  development  of  too 
many  major  weapons  and  support  systems.  Project  definition  has 
been  achieved  sooner  or  later  on  all  successfully  completed  develop¬ 
ment  projects  in  the  past  but  the  object  of  the  project  definition 
phase  is  to  achieve  this  sooner  rather  than  later  and  avoid  the 
disruptions  in  schedules,  increases  in  cost  and  losses  in  opera¬ 
tional  effectiveness  that  result  from  changes  caused  by  tardiness 
in  project  definition. 

An  objective  of  the  PDP  we  are  immediately  and  primarily  con'- 
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cerned  with  is  the  total  system  trade-offs  which  should  be  conducted 
during  PDP.  I  would  like  to  quote  from  the  DOD  directive  on  PDP 
concerning  these  trade-offs  which  states:  Trade-offs  shall  be 
used  to  obtain,  within  the  mission  and  performance  envelopes,  an 
optimum  balance  between  total  cost,  schedule,  and  operational  effective¬ 
ness  for  the  system.  In  this  context,  total  cost  means  the  total 
cost  of  acquisition  and  ownership  which  includes  development, 
production,  deployment,  operation,  and  maintenance;  operational 
effectiveness  includes  all  factors  influence  g  effectiveness  in 
operational  use  such  as  performance  capability,  reliability  and 
maintainability;  and  system  includes  the  hardware  itself  and  all 
other  required  items,  such  as  facilities,  personnel,  data,  training 
equipment,  etc.  I  think  these  statements  adequately  sum  up  what  we 
are  attempting  to  accomplish  in  obtaining  total  system  effectiveness. 

Of  further  significance  is  the  fact  that  PDP  studies  are 
usually  conducted  by  two  or  more  contractors  on  a  competitive  basis 
for  the  prize  of  a  multi-million  dollar  development  contract  if  a 
full  scale  development  is  directed  by  DOD  at  the  conclusion  of  the 
PDP.  This  competitive  aspect  of  the  PDF  has  the  effect  of  producing 
thorough  and  complete  trade-off  studies  which  are  considered  so 
important  at  this  point  in  the  development  process.  If  nothing 
else,  it  permits  you  to  help  us  keep  your  competitors  honest. 

The  PDF  can  logically  be  conducted  any  time  subsequent  to  the 
establishment  of  a  requirement  for  a  system.  This  vugraph  shows 
a  simplified  diagram  of  the  Navy  RDTS-E  planning  process  and  where 


-6- 


r  SPECIFIC 
OPERATI ONAL 
REQUIREMENT 
;  (SOR)  , 


the  PDP  would  fit  in.  The  process  shown  is  not  necessarily  classical 
and  specific  steps  could  vary  somewhat;  however,  the  basic  process 
is  representative. 

The  planning  for  an  effective  system  should  not  begin  with  the 
PDP  but  should  begin  with  the  initiation  of  the  development  planning 
process.  This  can  be  seen  by  reviewing  the  RDT&E  planning  process. 

RDT&E  planning  within  the  Department  of  the  Navy  is  character¬ 
istically  conducted  as  a  dialogue  between  the  user  interest  and  the 
producer  interest.  The  user  interest  is  represented  by  the  Chief 
of  Naval  Operations  and  the  Commandant  of  the  Marine  Corps,  as 
spokesman  for  the  operating  forces  and  the  producer  interest  is 
represented  by  the  Chief  of  Naval  Material  speaking  for  the  Naval 
Material  Support  Establishment.  This  user-producer  relationship 
is  more  analogous  to  a  relationship  between  cooperating  independent 
business  organizations  rather  than  to  traditional  military  relation¬ 
ships.  Parenthetically,  I  might  add  that  there  are  times  when  we 
wonder  if  the  analogy  should  be  labor-management  negotiation  rather 
than  buyer  and  seller.  Plans  are  the  result  of  negotiation  between 
the  two  interests.  Through  this  process  the  trade-offs  should  be 
made  which  will  result  in  the  maximum  military  capability  for  the 
Operating  Forces  possible  within  the  limits  of  the  resources  avail¬ 
able  to  the  Naval  Establishment. 

The  principal  documents  used  in  the  user-producer  dialogue 
are  shown  in  the  vugraph  as  the  intermediate  points  in  the  flow 
diagram. 

At  first  glance  the  impression  is  that  the  user  interest  levies 
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unilateral  requirements  —  based  on  pure  military  necessity  —  on 
the  producer  interest.  The  actual  process,  however,  involves  a 
continuous  interaction  between  operational  requirements  and  their 
spokesman,  and  technical  and  scientific  possibilities  and  their 
spokesman.  It  is  one  continuing  iterative  interchange.  New  formal 
requirements  for  weapons  hardware  more  often  than  not  have  their 
genesis  in  new  possibilities  stemming  from  advancing  knowledge 
and  technology  rather  than  from  evolving  military  need  or  reassess¬ 
ment  of  old  needs.  These  are  the  classes  which  Adm  Ruckner  in  his 
remarks  yesterday  tagged  as  "pushed"  operational  requirements. 

The  Chief  of  Naval  Operations  is  responsible  for  the  preparation 
of  a  General  Operational  Requirement  (GOR)  for  each  functional 
warfare  and  support  area.  GOR’s  usually  result  from  rather  extensive 
long  range  strategic  and  tactical  studies.  These  documents  state, 
in  relatively  broad  but  significant  terms,  the  capabilities  the 
Navy  needs  within  each  area.  For  guidance  in  making  trade-offs  in 
v:eapons  design  the  GOR  shoulc  contain  information  on  the  relative 
importance  of  the  various  capabilities  desired.  In  the  past, 
performance  specifications  have  been  adequately  stated  in  the 
GOR's,  however,  other  considerations  which  comprise  syste  1  effective¬ 
ness  —  reliability,  maintainability,  etc.  —  have  not  always  been 
given  adequate  attention.  Total  system  effectiveness  and  planning 
guidance  for  the  total  system  must  be  provided  as  feasible.  This 
is  the  beginning  of  system  effectiveness  planning  that  I  spoke 
of  earlier.  Here  is  where  we  start  thinking  and  planning  for 
total  system  effectiveness. 
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The  nex^|  step  in  the  RDT&E  planning  process  is  the  producer 
response  to  the  GOR  in  the  form  of  a  Proposed  Technical  Approach 
(PTA).  PTA's  are  developed  by  the  Naval  Material  Support  Establish¬ 
ment  to  propose  technically  feasible  alternative  methods  of  accomplish 
ing  objectives  set  forth  in  a  GOR.  The  PTA  should  be  fully  respon¬ 
sive  to  the  GOR,  therefore,  the  quality  of  the  PTA  depends  directly 
on  the  quality  of  the  GOR.  In  addition  to  numerous  other  mandatory 
requirements  of  the  PTA  which  are  not  of  particular  interest  here, 
the  governing  OPNAV  and  DOD  Directives  require  that  the  PTA  should, 
to  the  extent  that  it  can  be  determined  or  estimated,  analyze  and 
compare  the  operational  effectiveness  of  the  proposed  alternate 
development  approaches  in  terms  of  performance,  reliability,  operabi¬ 
lity,  and  maintainability  and  clearly  indicate  the  basis  of  the 
comprris ion ^ such  as  previous  experiments,  extrapolation  or  conjecture. 

The  user  side  of  this  dialogue  then  reviews  what  is  presented 
in  the  PTA  and  makes  a  decision  whether  or  not  to  pursue  further 
study  of  the  basic  requirement.  If  further  study  is  deemed 
appropriate  a  tentative  specific  operational  requirement  (TSOR) 
is  issued  to  the  producers  which  directs  initiation  of  a  study  effort 
prerequisite  to  the  establishment  of  a  development  program  to 
attain  the  capability  stated  in  the  TSOR.  The  TSOR  states  the 
need  for  achieving  a  particular  operational  capability  and  outlines 
the  identifiable  characteristics  necessary  in  a  warfare  system  to 
fulfill  the  requirement.  The  TSOR  defines  the  desired  performance 
goals  and  provides  additional  information,  such  as  the  plans  for 
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use,  needed  to  weigh  alternatives  and  make  the  trade-offs  required 
to  achieve  an  optimum  system. 

For  major  projects  meeting  the  threshold  requirements  I  mentioned 
a  few  moments  ago,  the  study  required  by  the  TSOR  usually  takes  the 
form  of  a  Project  Definition  Phase.  During  the  course  of  this  study 
a  Preliminary  Technical  Development  Plan'  and  a  Specific  Operational 
Requirement  (SOR)  are  evolved. 

The  most  important  end  product  of  the  PDP  is  the  Technical 
Development  Plan  or  TDP.  The  TDP  comprises  the  grand  plan  for 
the  fulfillment  of  the  requirements  as  originally  spelled  out  in 
xht  operational  requirements  of  the  user.  It  is  a  complete  and 
detailed  description  of  the  effort  necessary  to  accomplish  the 
development.  The  goal  of  a  TDP  is  a  balanced  and  integrated 
effort  aimed  at  optimizing  operational  effectiveness,  total  cost, 
and  early  availability. 

With  the  formulation  of  the  TDP  at  the  termination  of  the 
project  definition  phase  the  necessary  total  system  planning 
for  the  full-scale  development  phase  of  the  system  is  for  all 
intents  and  purposes  completed,  however,  during  the  full-scale 
development  phase  the  TDP  should  be  updated  as  required  but  if 
our  planning  has  been  adequate;  necessary  updating  will  be  at  a 
minimum. 

The  planning  process  leading  to  system  development  is  well 
outlined  with  the  requirements  and  glidelines  covering  the  documen¬ 
tation  required  adequately  defined  in  DOD  and  Navy  Department 
directives.  However,  these  directives  alone  do  not  insure  that 
the  guidelines  will  be  followed  and  the  requirements  fulfilled  in 
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planning  the  development  of  a  3ystem.  To  insure  that  requirements 
are  met  and  that  all  elements  of  Systems  Effectiveness  receive 
thorough  attention  and  adequate  consideration  by  the  Navy  Material 
Support  Establishment  the  Systems  Effectiveness  Branch  of  the 
Office  of  Naval  Material  analyzes  and  appraises  all  Proposed 
Technical  Approaches  and  Technical  Development  Plans. 

Operational  Requirements  originated  by  GNO  also  receive  a 
penetrating  review  by  this  Branch  to  insure  that  the  effectiveness 
requirements  for  the  system  are  adequately  included  in  these 
documents . 

Unfortunately,  this  System  Effectiveness  Branch  only  recently 
came  into  being  with  the  establishment  of  the  Chief  of  Naval  Material 
in  December  of  1963.  Since  that  time,  however,  it  has  been  the 
aim  of  this  Branch  to  exert  the  leadership  and  guidance  necessary 
to  provide  an  effective,  cohesive  effort  in  the  Navy  Material 
Support  Establishment  towards  systems  effectiveness.  With  ex¬ 
perience  this  leadership  and  guidance  will  become  even  more 
effective  since  the  beneficial  effect  of  cross-pollination  among 
the  various  projects  will  be  realized. 

As  you  can  see  there  are  now  requirements  and  procedures  inherent 
in  the  development  planning  process  designed  to  insure  that  total 
systems  effectiveness  is  planned  into  the  system  under  development. 
More  importantly  this  planning  is  now  forced  to  occur  early  in  the 
development  process  which  should  insure  that  the  desired  system 
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effectiveness  is  designed  and  built  in  to  the  system.  We  must 
remember,  however,  that  it  takes  people  to  implement  the  management 
procedures  involved  and  needless  to  say  the  success  of  any  management 
process  is  directly  dependent  upon  these  people.  These  people,  and 
I  refer  to  the  people  of  industry  as  well  as  Navy  people,  must  be 
oriented  towards  and  dedicated  to  the  total  system  effectiveness 
approach  to  development  planning  and  net  to  any  one  element 


of  this  discipline  in  the  manner  pure  performance  capability  was 
in  the  past.  We  must  also  avoid  including  elements  such  as 
maintainability  for  maintainability' s  sake  or  reliability  for 
reliability's  sake  just  because  its  required  or  because  its  the 
vogue.  Each  of  these  elements  must  be  weighed  and  carefully  traded- 
off  for  each  particular  system  under  development.  The  weights  ap¬ 
plied  to  these  elements  will  of  course  have  to  vary  between  systems 
in  order  to  appreciate  the  different  functions  of  these  various 
systems . 

Of  more  significance  than  the  fact  that  these  weights  vary  is 
the  nature  of  the  control  of  the  variance.  Although  complex  in 
detail,  the  general  rule  is  readily  stated.  How  much  value  is 
t..-.s  -unc„icn  to  cLCCOIupi  jLOiiluGn't  cf  the  system' s  mission  as 
against  the  other  system  functions?  The  answer  to  this  question 
provides  the  weighting  criteria.  In  our  plannings  as  indeed  all  of 
our  efforts  in  the  military-can  have  but  one  end  objective  -  the 
military  mission.  It  therefore  is  necessary  that  our  planning  have 
as  its  focus  the  military  mission  of  the  system.  This  requires  that 
our  planning  weighting  factors  also  be  determined  by  mission  consi — 


derations. 


GOST  FACTORS  IN  SYSTEMS  DESIGN 
PRESENTED  AT  THE 
NORTHEASTERN  STATES 
NAVY  RESEARCH  AND  DEVELOPMENT  CLINIC 
PHILADELPHIA,  PENNSYLVANIA 
19  NOVEMBER  I964 


Mr.  John  W.  Stone 
Office  of  Naval  Material 
Department  of  the  Navy 


I  would  like  to  discuss  some  of  the  factors  relating  to  cost 
effectiveness  and  the  influences  of  these  factors  on  systems  design. 

The  previous  papers  have  discussed  various  important  elements  of  system 
effectiveness,  all  of  which  relate  to  and  affect  one  another.  It  is 
this  inter-relationship  of  these  various  elements  which  form  -the  basis 
for  naval  systems  decision  making.  Cost  is  another  major  element  to  be 
considered  in  the  decision  making  process.  Cost  is  probably  the  one 
element,  or  ingredient,  when  properly  weighed,  becomes  a  yardstick  which 
will  bring  all  elements  into  proper  focus. 

Webster  defines  "COST'1  as  "THE  LOSS  OR  PENALTY  INCURRED  IN  GAINING 
SOMETHING."  Cost  is,  therefore,  an  exchange  or  trade-off.  It  is  this 
trade-off  connotation,  which  is  the  key  or  measure  used  in  determining 
effectiveness. 

System  Effectiveness  has  been  defined  as  "THE  PROBABILITY  THAT  THE 
SYSTEM  WILL  OPERATE  SUCCESSFULLY  WHEN  REQUIRED  UNDER  SPECIFIED  CONDITIONS 
OR  ENVIRONMENT."  Mr.  Rohe,  in  his  discussion  of  engineering  integration, 
expressed  the  term  in  its  gross  sense  by  using  the  formula:  Es  =  PAU 
(Figure  l) .  "E  ",  system  effectiveness,  being  the  product  of;  "P",  the 
performance  or  system’s  capability;  "A",  the  availability  or  the  fraction 
of  time  the  system  is  ready  and  capable  of  performing  its  mission;  and 
"U",  the  utilization  or  fraction  of  the  performance  capability  actually 
utilized  for  a  specific  application  in  a  specified  environment.  Therefore, 
an  increase  in  one  or  more  of  the  elements  of  performance,  availability 
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ur  utilization  will  have  an  attendent  increase  in  the  effectiveness  of 
the  system.  An  increase  in  systems  effectiveness  means  we  have  gained 
something.  For  this  gain  there  has  also  necessarily  been  corresponding 

loss  or  penalty  incurred  -  a  cost.  What  are  these  costs  and  how  do 

they  affect  the  system?  This  provides  the  alternatives  or  trade-offs, 
which  are  the  inputs  required  by  the  decision  maker. 

The  injection  of  cost  provides  a  new  dimension  to  the  term  systems 
effectiveness  and  establishes  it  as  more  than  a  sterile,  academic  exercise 
To  distinguish  this  new  discipline,  we  can  term  it  cost  effectiveness. 

Cost  effectiveness  can  be  defined  simply  as  the  ratio  between  systems 
effectiveness  and  its  attendant  costs.  Expressed  in  gross  terms  the 
formula  is: 

Ec  =  fT?C  (Figure  2) 
a  u 

Ec  being  cost  effectiveness  which  is  the  previously  defined  systems 
effectiveness  (the  product  of  £erformance,  availability  and  utilization) 
divided  by  the  total  cost.  The  total  cost  is  expressed  here  as  the  sum 
of  C&,  the  cost  of  acquisition,  plus  Cu,  the  cost  of  utilization  or  as 
it  is  sometimes  referred  to  as  the  cost  of  ownership 

The  denominator  could  just  as  well  have  been  expressed  as  total 
program  cost;  however,  I  have  separated  the  total  cost  into  these  two 
broad  categories  to  ennhasize  the  need  to  insure  that  all  costs  are 
considered.  Figure  3  illustrates  what  is  included  in  these  categories. 
Cost  of  acquisition,  as  stated  before,  is  the  total  dollar  cost  of 
development  and  production.  Development  costs  include  all  dollar  costs 
associated  with;  operation  analysis,  system  definition,  system  design. 
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EXTERNAL  COSTS  DUE  TO  FAILURES 


hardware  design,  test,  and  evaluation.  Production  costs  include  all 
dollar  costs  associated  with;  procurement,  manufacture,  installation,  test, 
and  training. 

Cost  of  utilization  is  the  average  annual  dollar  cost  of  operating 
and  maintaining  the  system,  including  the  external  cost  of  its  failures, 
multiplied  by  the  number  of  years  of  useful  life.  Operational  cost  is 
the  long  term  annual  dollar  cost  of  system  for  operating  personnel, 
facilities,  utilities  and  special  inputs  required.  Maintenance  cost  is 
the  long  term  annual  dollar  cost  of  system  maintenance  personnel, 
facilities,  spare  components,  logistics  and  diagnostic  aids,  etc.  Also, 
we  must  not  forget  the  costs  which  are  external  to  the  system  but  are  as 
a  consequence  of  system  failures. 

By  proper  emphasis  of  all  cost  elements,  the  total  cost  becomes 
meaningful  to  the  decision  makers.  Too  often  decisions  concerning  new 
o/stems  have  been  made  considering  only  the  estimated  cost  to  develop  and 
procure.  With  our  eye  focused  on  this  cost  of  acquisition  we  find  our 
astigmatism  has  blurred  the  cost  of  utilization  aspect.  From  a  develop¬ 
ment  and  production  standpoint  the  x'st  effectiveness  of  a  particular 
system  may  be  outstanding.  However,  without  adequate  attention  to  the 
cost  of  utilization  the  Fleet  can  find  itself  with  a  system  which  is 
plagued  with  excessive  operating  and  maintenance  costs.  Corrective  action 
is  usually  expensiye  and  more  often  than  not  the  system  will  still  have 
a  greatly  reduced  effectiveness.  Such  a  case  can  be  the  result  of  the 
cost  effectiveness  not  including  all  the  trade-off’s  necessary  to  make 
the  proper  decision.. 


I  would  like  to  illustrate  by  setting  up  a  hypothetical  example. 

Fcr  '■his  example,  I  have  chosen  two  similar  systems  and  labeled  them 
s  -tern  "x"  and  system  "yn,  respectively.  If  we  ignore  the  cost  context, 
the  System  Effectiveness  is  determined  by  the  formula  Eg  =  PAU.  (System 
Effectiveness  being  the  product  of  Performance,  Availability  and  Utili¬ 
zation.)  Figure  4  shows  the  System  Effectiveness  without  the  element 
of  cost  as  being  46O  for  system  "x"  and  612  for  system  "y".  It  is,  of 
course,  assumed  that  the  calculations  for  both  systems  are  to  a  common 
base  of  total  mission  capability  of  1000.  Also  let  us  assume  the 
unlikely  situation,  where  the  confidence  factors  are  equivalent. 

Therefore  from  analysis  of  the  system  effectiveness  we  would  elect 
system  "y"  because  of  the  margin  612  has  over  4&0- 

Let's  look  at  what  happens  when  we  consider  the  cost  effectiveness 
of  only  cost  of  acquisition.  To  do  this  we  ratio  the  effectiveness  of 
systems"x"  and  "y"  by  the  cost  of  acquisition.  As  I  stated  before,  cost 
of  acquisition  is  the  tota]  dollar  cost  of  development  and  production. 
Going  back  to  our  hypothetical  example,  the  cost  of  acquisition  for 
system  "x"  is  $5. CM  while  for  system  !,y"  it  is  $6.CS-I.  The  penalty  to 
acquire  the  additional  capability  of  system  "y"  over  system  "x"  is  seen 
to  cost  $1.CM.  Figure  5  ratios  the  previous  "x"  and  ,!y"  values  f  system 
effectiveness  by  their  respective  costs  cf  acquisition.  The  resulting 
cost  effectiveness  has  an  index  of  92  for  system  "x"  and  102  for  "y". 

Cost  effectiveness  analysis,  from  the  cost  of  acquisition  standpoint, 
would  also  select  system  "y"  as  being  superior;  as  diu  the  previous  system 


effectiveness  model. 
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Up  to  Here  we  have  neglected  our  cost  of  utilization  or  cost  of 
ownership.  Cost  of  utilization,  as  given  before,  is  the  average  annual 
dollar  cost  of  operating  and  maintaining  the  system,  including  the  external 
cost  of  its  failures,  multiplied  by  the  number  of  years  useful  life. 

Let's  take  our  hypothetical  "x"  and  "y"  systems  and  see  the  effect 
the  cost  of  utilization  has  on  cost  effectiveness.  System  "y"  with  its 
superior  capacity  has  a  cost  of  utilization  of  $8M.  System  "x"  has  a 
cost  of  utilization  of  $4M.  Figure  6  incorporates  these  values  into  the 
cost  effectiveness  formula  we  find  system  ”y”  with  an  index  of  43.7  while 
system  "x"  enjoys  a  larger  value  of  51.1.  By  considering  the  total  cost, 
we  find  the  system  with  the  better  cost  effectiveness  is  system  "x",  not 
system  "y"  as  previously  indicated. 

This  is  a  very  crude  example.  However,  as  ever  increasing  require¬ 
ments  are  placed  on  us  to  design  systems  with  quantum  jumps  in  systems 
performance,  we  cannot  overlook  any  of  the  factors  of  cost.  We  must  not 
only  ask  ourselves  if  we  can  afford  the  system  from  the  development  and 
production  aspect,  but  also  can  we  afford  the  cost  of  ownership.  The  old 
saying  "It  is  not  the  initial  cost  but  the  upkeep"  is  just  as  true  for 
naval  systems  as  anywhere  else. 

We  are  accustomed  to  thinking  of  cost  as  being  dollars.  However, 
dollars  are  only  the  medium  of  exchange  and  as  such  dollars  are  not  true 
resources.  Commander  Sargent  expresses  cost  as  having  four  real  coins. 

The  four  real  coins  of  cost  are: 

(1)  Manpower 

(2)  Material 

(3)  Facilities 

(4)  Time 


There  is  a  degree  of  exchange  available  among  these  four  coins  or 
resources.  The  optimization  of  effectiveness  must  take  cognizance  of 
these  available  trade-offs.  Cost  effectiveness  is  a  measure  of  how  well 
we  spend  these  four  real  coins  of  cost  for  the  purpose  of  gaining  in 
system  effectiveness. 

To  conclude,  I  would  like  to  point  out  that  all  of  the  aspects  of 
cost  are  under  the  cognizance  of  the  systems  designer,  whom  I  enjoin  to 
insure  that  proper  consideration  is  given  to  all  of  the  cost  factors. 

Only  by  so  doing,  are  the  necessary  inputs  available  for  proper  system 
decisions.  We  must  have  systems  which  have  a  reasonable  cost  of  ownership. 
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MAN  PARAMETERS  IN  SYSTEM  SUPPORT 


In  recent  years  we  in  the  Armed  Forces  have  been  referring  to' 
our  warfare  systems  as  man-machine  systems.  That  we  have  combined 
men  and  machines  in  warfare  has  been  true  since  man  discovered  the 
effectiveness  of  the  simple  lever  machine  called  a  club.  Why  -  then  - 
do  we  use  the  hyphenated  expression,  man-machine,  now  as  though  the 
combination  were  a  new  c’scovery?  Is  there  really  something  new 
or  is  this  simply  a  change  in  mode  of  expression? 

I  would  submit  that  there  is  indeed  something  very  new  and 
different  which,  although  subtle  and  not  always  fully  understood  by 
the  users  of  the  expression,  "man-machine  systems",  is  fundamental 
and  MUST  be  thoroughly  understood  by  all  connected  with  the  military 
development  process,  particularly  those  decision-makers  to  whom  we 
sometimes  loosely  refer  as  management. 

During  the  days  of  club,  sling,  spear  and  arrow  each  fighter  had 
his  own  machine.  This  machine  was  t  ssentially  a  direct  extension 
of  the  individual's  capability.  With  it  he  could  hit  harder  and 
further  than  he  could  without  it.  As  warfare  technology  evolved,  range 
and  hitting  power  increased.  The  machine  changed  from  the  simplest 
level  to  ever  more  complex  devices  often  requiring  more  than  one  man 
to  operate.  This  evolutionary  process  continued  along  the  same  track 
until  ranges  exceeded  man's  capability  to  put  the  machine  on  target 
with  the  unaided  eye.  It  therefore  became  necessary  to  extend  his 
ocular  capability  by  the  use  of  optical  systems.  Even  as  late  as  the 
early  days  of  World  War  II  -  machines,  as  complicated  as  they  were  with 


respect  to  the  war  club,  were  still  ju6t  extensions  of  man-s  capabi¬ 
lity  to  see  and  to  strike.  Stimuli  were  perceived  by  a  man  and  he 
Initiated  the  strike.  The  very  control,  although  sometimes  machine 
assisted,  was  still  done  through  man's  own  motor  control  mechanisms. 

Two  significant  technological  innovations  in  our  time  are  changing 
the  basic  evolutionary  pattern  which  has  existed  since  before  re¬ 
corded  history.  I  say  "are  changing"  because  their  full  impact  has 
not  yet  been  felt.  Indeed,  early  applications  were  in  the  context 
of  the  old  idea  of  being  used  simply  as  an  extension  of  man's  capa¬ 
bility.  The  first  of  these,  radar,  illustrates  my  point  quite  well. 

Its  very  name,  radar,  was  derived  from  its  application,  radio  direction 
and  ranging.  We  were  merely  extending  our  ocular  capability  through 
electromagnetic  means  because  of  its  range  superiority  over  optical 
means . 

Although  not  generally  recognize!,  the  birth  of  the  new  era, 
symbolized  by  the  term  man-machine  systems,  occured  when  the 
technologists  produced  the  automatic  tracking  and  fire  control  or  gun 
laying  radars.  This  was  qulc-.ry  followed  by  the  second  technologi¬ 
cal  innovation,  high  speed  electronics  computers.  With  these  machines 
we  hod  devices  which  could  replicate  the  logic  process  whicn  hereto¬ 
fore  had  beer,  the  exclusive  domain  of  man.  No  longer  were  machines 
simply  the  sensory  and  physical  extensions  of  man. 

With  the  invasion  by  t.-.e  machine  into  the  logic  process,  hitherto 
exclusively  man's,  the  relative  roles  of  man  and  machine  have  under¬ 
gone  a  subtle  but  nonetheless  fundamental  change.  No  longer  can  we 
regard  man  as  an  entity  apart  from  the  system  -  an  entity  who  operates 


maintains  or  controls  the  machine.  Bather  he  is  explicitly  a  part 
of  the  system  contributing  those  capabilities  which  are  urfiquely 
his.  Thus  in  theory  at  least  we  now  have  man-machine  systems  with 
the  man  assigned  those  tasks  which  he  can  do  most  effectively  and 
and  efficiently  and  the  machine  assigned  those  tasks  which  it  can 
do  most  effectively  and  efficiently. 

This  nice,  logical  rationale  provides  a  philosophy  for  our 

systems  concepts.  There  remains  r>~  problem  *•-  implementation. 

2 

Einstein’s  equation  is  sheer  simplicity,  E  =  MC  .  But  the  problems 
attendant  to  the  exploitation  and  implementation  of  the  concept 
represented  by  this  simple  equation  are  too  well  known  to  be  recited 
here.  It  has  been  said  that  simple  solutions  stem  only  from  simple 
problems.  Like  the  Einsteinian  concept,  ours  poses  anything  but  a 
simple  problem. 

I  shall  not  attempt  to  discuss  the  whole  problem.  Rather  I’d 
like  to  confine  myself  to  one  aspect  of  the  problem,  man  in  the 
system. 

Much  as  it  may  bruise  individual  egos,  man  subject  to  the 
machine  even  as  the  machine  is  to  him  through  the  interactions  which 
take  place  in  today’s  complex  systems.  Certainly  man's  judgement 
must  prevail  and  in  a  sense  can  be  considered  to  control  since  the 
machine  does  not  possess  intellect.  However,  we  must  not  lose  Bight 
of  the  fact  that  even  this  "man-only"  attribute  can  be  and  is  influenced 
to  a  remarkable  extent  today  by  the  method  of  processing  and  manner 
of  display  of  the  processed  data  by  the  machine. 

Quite  apart  from  considerations  of  force  levels  or  technological 
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advances,  the  realities  of  man-machine  interactions  dictate  a  much 
harder  and  more  studied  approach  to  the  man  and  his  functioning  in 
the  system.  Lest  I  mislead  you,  let  me  make  it  clear  that  I  am  not 
referring  solely  to  the  lower  skill  levels  who  obviously  fulfill  the 
transfer-function  between  segments  of  the  system.  I  refer  also  to 
the  Commander.  Due  to  his  displacement  from  the  data  .source,  he  is 
actually  subject  to  incremental  machine  decisions  to  a  greater  extent 
than  those  who  are  lower  in  the  hierarchy  of  decision  making.  There¬ 
fore  he  must  be  assured  that  the  maximum  capability  of  both  man-segments 
and  machine-segments  are  brought  to  bear  by  way  of  providing  the 
best  possible  inputs  to  his  level  of  function  in  the  system.  As  we 
face  threats  of  ever  higher  velocities,  this  becomes  the  more 
pressing  since  the  Commander  has  a  diminishing  reaction  time  in  which 
tc  make  his  decision.  He  thus  becomes  more  dependent  upon  the  in¬ 
cremental  decision-making  at  lower  echelons  in  the  system. 

In  those  echelons  which  are  purely  machine,  the  task  of  valida¬ 
ting  relatively  easy.  We  know  the  machine  can  make  only  those 
decisions  which  a  man  has  made  previously.  Further  these  were 
carefully  checked  and  rechecked  in  an  environment  apart  from  battle 
stress.  But  —  these  machine-segments  have  an  Achille’s  heel.  In  the 
computer  fraternity  they  pithily  expres  s  it  as  arbage  in  -  garbage 
out".  No  matter  how  valid  the  programming,  the  quality  of  output 
can  be  no  better  than  that  of  the  input. 

Frequently  these  inputs  come  from  the  man-segments.  These  are 
much  more  difficult  to  validate.  We  must  consider  both  the  quantita¬ 
tive  and  qualitative  variances  in  performance  among  the  specific  individuals 
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who  may  be  assigned  the  task  from  time  to  time  as  well  as  the  variances 
of  given  individuals  in  time.  To  add  to  the  problem,  we  are  woefully 
lacking  in  the  means  for  measuring  the  performance  and  capabilities 
of  the  man.  Neither  do  we  know  enough  about  how  and  why  he  functions. 
We  cannot  give  our  system  designers  anything  approaching  the  complete¬ 
ness  of  the  capability  description  we  provide  for  the  machine  seg¬ 
ments  he  uses.  This  area  of  effort,  termed  Human  Factors  Engineering, 
must  be  greatly  expanded  from  present  effort  levels  if  we  are  to 
achieve  maximum  systems  effectiveness. 

Within  the  context  of  our  philosophical  concept  Human  Factors 
Engineering  is  a  very  broad  area  of  concern  encompassing  evch  diverse 
disciplines  as  the  full  gamut  of  the  behavioural  sciences,  phslology, 
anthropometries  and  psychometrics  in  the  research  and  development 
sphere.  In  an  applied  Engineering  sense,  it  Includes  the  area  which 
we  have  traditionally  referred  to  as  Itersonnel  Management  end  Training. 
Actually,  one  can  conceive  of  the  Itersonnel  and  Training  people  as 
being  the  producers  of  the  man-modules  for  our  systems.  It  is  to 
them  that  our  systems  engineers  look  for  the  man  in  the  systems.  It 
ia  to  them  also  that  the  systems  engineers  look  for  the  descriptive 
specifications  of  the  man  available  for  incorporation  into  the 
system. 

Herein  lies  the  problem.  While  there  is  a  positive  effort  to 
provide  quality  control  in  processing  the  product  and  in  selection 
of  the  raw  material  Input,  the  random  and  parenthetically  frequently 
accidental  nature  of  the  origins  of  the  raw  material  poses  real 
difficulties.  As  a  result,  descriptive  specifications  are  given  in 
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very  broad  parameters. 

This  situation  Is  aggravated  by  our  lack  of  real  understanding 
as  to  bov  and  vfay  this  raw  Bate  rial,  man,  functions,  neither  do  ve 
have  the  attendant  ■ensuring  systems  for  this  functioning.  Ve 
point  with  pride  to  the  fine  tolerances  to  which  ve  can  produce 
machine -elements .  Ve  Measure  then  with  micron  exactness.  Then  ve 
ask  the  system  designer  to  combine  them  with  nan-elements  which  ve 
describe  as  an  average  aan  with  an  8th  grade  mentality.  What  preci¬ 
sion!  What  an  exquisitely  defined  measurement  scale  I  —  and 
management  says,  "Give  us  systems  effectiveness." 

That  ve  seed  systems  effectiveness,  particularly  in  the 
military  in  these  days  of  hypervelocity  weapons,  is  a  reality  that 
is  ineontrovertable.  However,  the  achievement  of  system  effectiveness 
is  highly  problematical  until  and  unless  ve  can  solve  the  problems  of 
the  man  parameters  in  cystem  support.  Our  technology  has  reached  a 
noint  where  ve  may  no  longer  hide  our  relative  Ignorance  of  how  and 
why  man  functions  behind  the  idea  of  the  adaptability  of  man.  That 
ve  are  straining  the  boundaries  of  this  adaptability  is  evidenced 
t-Y  the  Incidence  of  cardiac  and  neurasthenic  casualties  among  our 
military.  It  is  also  evidenced  by  the  fact,  as  reported  by  the  FAA, 
that  80#  of  our  aircraft  accidents  are  attributable  to  pilot  error. 

We  Must  stop  using  adaptable  man  as  the  multi-purpose  gap-filler 
between  machine s.  Ve  must  strive  for  the  design  goal  which  uees  man 
w.'th  all  deliberateness  to  perform  those  functions  in  which  he  Is 
superior  to  the  machine  rather  than  to  perform  those  functions  which 
ve  don't  know  how  to  design  a  machine  to  perform,  often  with  little 
regard  to  either  the  level  of  quality  or  the  quantity  of  tasks 
assigned. 
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In  order  to  reach  the  design  goal  we  must  first  learn  far 
more  than  ve  now  know  about  how  and  why  a  nan  functions.  We  must 
learn  how  to  Measure  the  parameters  which  describe  these  functions. 

We  oust  acquire  the  capability  to  describe  exactly  what  combinations 
of  nan-functions  are  (or  potentially  are)  in  our  available  Inventory 
together  with  the  distributions  of  these  functions.  Until  ve  are 
able  to  provide  adequate  nan  parameters  to  our  systems  designers, 
the  probabilities  of  true  systems  effectiveness  will,  continue  to 
be  quite  low  and  high  systems  effectiveness  will  be  more  accidental 
than  calculated.  Low  systems  effectiveness,  I  submit,  is  the  situa¬ 
tion  today.  Although  not  so  reported,  this  is  aa;.lfested  In  myriad 
reports. 

There  reports  use  such  terms  as  "too  complicated  machines," 
"inadequate  training,"  "above  the  heads  of  our  people,"  "can't  be 
maintained,"  etc.  These,  if  you  will,  are  symptoms.  These  demonstrate 
our  inability  to  fit  the  available  man-modules  to  the  system.  I 
would  offer  as  evidence  of  thlr  contention  that,  in  almost  every  case, 
we  are  able  to  provide  a  combination  of  man-modules  and  machine -modules 
that  does  function  effectively.  More  often  than  not  we  ascribe  tbe 
difference  between  tbe  successes  and  failures  to  such  things  as 
leadership,  luck  or;  In  some  cases,  a  unique  set  of  circumstances. 

In  any  case,  one  effective  system  case  is  evidence  that  the  system 
can  work  effectively. 

If  you  will  think  back  on  the  systems  each  of  you  has  been 
concerned  with,  I  believe  you  can  see  a  ciiaracterlstic  pattern.  In 
our  egocentric  way,  we  attribute  our  successes  to  the  quality  of 
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the  men  in  the  system  and  our  failures  to  the  inadequacies  of  the 
machine  in  the  system.  We  explain  the  "why"  of  the  machine  inade¬ 
quacies  quite  exactly.  But- -do  we  explain  the  "whys"  of  our  man- 
attributed  successes  with  the  same  exactness?  Of  course  we  don't. 

We  don’t  know  how.  "The  leadership  was  better."  "A  certain 
technician  knew  more."  How  much  better?  How  much  more?  These 
are  characteristics  of  our  man-machine  systems  that  we  haven't  yet 
learned  how  to  quantise.  In  the  absence  of  this  capability  we 
are  constrained  *,o  use  comparative  verbiage  which  is  singularly 
ucpreclse  and  subject  to  value  changes  in  the  communication  process. 

Quantizing  the  man  function  then  becomes  the  core  problem 
to  our  systems  effectiveness  effort.  What  do  we  do  about  the  Man 
Rarameters  in  System  oupport?  To  me,  the  solution  is  quite  clear. 
First,  management  -  and  explicitly  Armed  Forces  Management  -  must 
admit  to  not  being  the  fount  of  knowledge  in  the  appraisal  and 
evaluation  of  the  man-function  -  admit  that  at  best  ve  are  using 
boilermaker  measurements  on  watch  movements.  Am  I  really  overstating? 
Look  at  the  three-element  0,  S  &  U.  system  ve  use  in  evaluating 
our  civil-service  personnel.  Look  at  theirs  or  our  own  Job-descrip¬ 
tions.  Look  at  Efficiency  Ratings  or  Fitness  Reports,  How  many 
of  the  latter  have  you  made  out  against  the  Job  requirements  or  billet 
description  much  less  against  any  empirical  performance  standard? 
Certainly  we  do  the  best  we  can  with  the  tools  we  have  available. 

But,  these  tools  don't  begin  to  appi'oxlmate  the  performance 
standards  and  measures  we  employ  for  machine -segments  of  our  systems. 

If  we  are  to  resolve  the  problem,  to  me  the  most  vexing  facing 
top  Military  Management  today,  we  must  undertake  a  program  of  study 
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lysis  of  the  man  parameters  in  systems  far  greater  than  that 
which  we  currently  have  underway.  We  must  close  the  gap  in  our 
understanding  and  measurement  of  the  man-parameters.  We  must  initiate 
and  support  efforts  in  scientific  study  which  will  lead  to  an  under¬ 
standing  and  measurement  of  the  man  akin  to  that  we  possess  for  the 
gear,  the  electron  or  red  fuming  nitric  acid. 

To  this  end,  a  number  of  projects  are  underway  In  the  Navy. 

In  an  attempt  to  get  a  handle  on  the  problem,  the  Bureau  of  Ships 
supported  by  the  Office  of  Naval  Material  has  initiated  a  project 
called  TRIM.  TRIM  is  u.»  acronym  for  Training  Requirements  Informa¬ 
tion  Management.  TRIM  is  a  systematic  approach  to  the  codification, 
recording  and  collection  oi  training  requirements  data  and  personnel 
resource  data  in  terms  of  training.  Rsrhapa  the  most  significant 
aspect  of  TRIM  is  that  its  design  concept  takes  into  account  the 
gross  nature  of  existing  measures  of  man  parameters.  As  a  result 
the  matrices  in  the  system  have  been  designed  to  provide  for  ulti¬ 
mately  more  refined  measures  without  necessitating  a  new  data  system. 

A  second  Navy  project,  which  I’d  like  to  cite  xe  the  effort 
under  the  sponsorship  of  the  Chief  of  Naval  Personnel  referred  to  as 
the  New  Developments  Human  Factors  Program.  This  is  a  rather  broad- 
gauged  effort  to  define  the  problem  and  provide  solutions  in  the 
personnel  management  and  training,  or  if  you  will,  production  processes 
for  our  man-aodul *s. 

In  a  more  penetrating  manner  but  with  somewhat  narrower  scope, 
the  Naval  Aviation  Development  Center,  Johnsville,  Pa.  and  the  School 
of  Aviation  Medicine  at  Psnsacola,  Fla.  have  been  probing  into  man- 


9 


\ 

measurements  peculiar  to  the  aerosphere  and  the  Naval  Medical 
Reseercb  Institute  at  Bethesda  has  done  similar  work  in  the  hydro¬ 
sphere.  These  latter  efforts  have  teen  dominantly  in  the  bio-medical 
field. 

NASA  and  Air  Force  have  probed  somewhat  more  deeply  into  the 
psychological  aspects  of  man's  behaviour  in  the  Mercury  and  APOLLO 
projects.  This  work  oas  proiuced  valuable  spin-offs  for  terrestrially 
limited  systems  but  Is.  focused  principally  at  a  highly  specialized 
situation  which  can  aff-rl  highly  specialized  men  and  environmental 
Ci  kitro**>s » 

However,  •  *  r-  *  bulk  *f  cur  -nilitary  systems  must  use  the 
so-callei  averaw  -nan.  Further,  nighjy  specialized  and  \  ery  expensive 
artificial  envir  umeots  are  simply  not  economically  feasible  for 
r/t  ■ec 

Th* refer®  -a  must  lear?  rr-  about  how  and  why  this  average 
pc. rf ,  -r.s.  V.*  must  learn  new  *  c  measure  and  predict  this  performance. 

r-»  r.v  »  tnese  ?r*i-Cti:ns  may  then  be  used  by  the  system 

designer  ■-  *.  the  :e«cr'_pt:v*  parar-ters  cf  the  man  in  the  system. 

Thee  anc  roly  ther  can  hope  to  achieve  overall  systems  effectiveness 
in  our  military  systems. 
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THE  KEY  TO  DEVELOPMENT  PAY  OFF 
REMARKS  OF 

REAR  ADMIRAL  E.  A.  RUCKNER 
AT  THE  SYSTEMS  EFFECTIVENESS  PAN  &L 
THE  WESTERN  STATES  NAVY  R&D  CLINIC 
KCNTANA  STATE  CC' LEGE 
BOZEMAN,  MONTANA 
22  July  1964 


Dr.  Trumbull,  members  of  the  Systems  Effectiveness  Fune!  of  the 
Western  States  Navy  R&D  Clinic,  ladies  and  gentlemen, 

The  subject  matter  of  this  session  represents  the  most  con?>rehensive, 
and  hence  most  critical,  problem  challenging  the  Research  and  Development 
community  today.  It  is  axiomatic  that  technological  advances  are  of 

little  use  if  the  systems  they  produce  are  not  effective.  As  the  Deputy 

0 

Chief  of  Naval  Material  for  Development,  my  concern  is  that  all  naval 
development  and  particularly  those  combinations  of  men  and  machinery 
which  we  term  warfare  systems  provide  to  our  fleet  a  maximum  capability 
in  fulfilling  their  national  defense  role.  This  capability  varies 
directly  with  the  systems  effectiveness  of  the  man-machine  combinations. 

Having  established  myself  as  being  for  motherhood  and  against  sin, 
let's  look  behind  the  platitudes  to  see  what  are  their  implications. 

Are  these  just  brave  words  —  or  —  is  there  a  key  which  will  open 
the  door  to  development  pay  off? 

If  you  have  inferred  from  the  foregoing  that  I  consider  that 
we  are  not  making  technological  progress,  disabuse  your  minds  of  the 
thought.  However;  if  you  infer  that  I  consider  that  we  are  not 
realizing  an  optimum  payoff  from  the  ever  accelerating  curve  of 
technical  progress,  you  are  quite  correct.  This  I  attribute  in  a 
large  measure  to  overextension:  -  overextension  of  the  men  in  the  system 
and  overextension  of  the  men  designing  the  system.  The  dangers  attendant 
to  overextension  are  not  unknown. 

History  is  full  of  examples  of  failures  and  defeats  caused  by 
people  who  were  in  too  much  cf  a  hurry.  Humans  have  a  tendency  to  become 


so  enchanted  with  progress  that  they  fail  to  recognize  the  dangers 
of  overextension.  Military  commanders  are  aware  of  the  necessity  to 
consolidate  their  positions  after  a  more  forward,  before  striking 
out  again  on  a  new  advance.  Chess  players  are  aware  of  the  fallacy 
in  moving  too  quickly  into  enemy  territory  without  adequate  consoli¬ 
dation  of  home  defenses.  Businessmen  can  testify  to  the  dangers  of 
over  committing  themselves  in  many  of  their  activities.  The  stock 
market  crash  of  1929  shows  what  happens  when  "progress*  is  not  based 
on  a  firm  foundation. 

To  say  to  an  audience  such  as  this  that  technology  is  advancing 
at  a  tremendous  rate  is  little  like  carrying  coeds  to  Newcastle. 

But  it  is  a  fact  which  we  must  not  overlook  despite  its  familiarity. 

In  an  effort  to  keep  pace  with  technology,  we  —  industry  and  the 
military  —  are  feverishly  extending  ourselves.  Even  though  people 
ARE  adaptable,  there  has  been  little  basic  change  in  them  over  the 
centuries.  Yet  we  are  being  called  upon  to  match  this  relatively 
unchanging  man  to  a  virtually  exploding  technology.  Adaptability 
alone  cannot  bridge  the  widening  gap  between  the  two  without  stretching 
the  people  beyond  their  tolerance. 

As  we  try  to  keep  pace  with  technology,  our  efforts  produce  ever 
more  complex  combinations  of  men  and  machines.  The  slope  of  the 
technological  capability  curve  far  exceeds  that  of  the  people  capbility 
curve.  The  result  is  overextension  of  the  *man  in  the  system*.  In 
addressing  ourselves  to  this  problem,  we  must  concentrate  on  correcting 
the  complexities  of  the  machine  segments  of  the  system  and  on  examina¬ 
tion  of  the  place  of  the  man  in  this  man-machine  system.  Failure  to  do 
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so  will  result  in  increasing  inability  to  obtain  sufficient  quanti 
ties  of  personnel  trainable  to  match  and  cope  with  the  complexities  of 
modern  warfare  material,  which  for  reasons  I  do  not  fully  understand, 
we  call  ■sophisticated*  systems. 

Sophisticates,  in  a  social  sense,  are  not  usually  associated  with 
the  production  of  worthwhile  results.  As  defined  by  Webster,  the  very 
sophisticate  is  to  deprive  of  genuineness,  naturalness,  or  simplicity; 
to  disillusion;  to  make  worldy-wise.  Yet  we  refer  to  these  complex 
systems  of  machines  and  overextended  men  as  sophisticated.  Could 
it  be  because  we  have  deprived  them  of  their  simplicity? 

Certainly  in  the  areas  of  missiles  and  electronics,  in  particular, 
our  fleet  systems  do  not  represent  simplicity.  To  say  that  these 
have  been  deprived  of  their  simplicity  is  difficult.  The  treats  which 
our  systems  must  counter  ARE  complex.  Solutions  to  the  problems  these 
threats  pose  are  complex.  But  this  does  not  give  license  for  complexi¬ 
ties  unlimited. 

Both  the  military  and  industry  have  contributed  complexities  which 
are  of  challengeable  merit.  Both  must  ask  themselves  the  question  *Is 
this  really  necessary  or  worthwhile?*  We  must  vigorously  resist  the 
introduction  of  additional  features,  functions  and  other  complicating 
mechanisms  which  are  not  vital  to  the  mission  effectiveness  of  the 
system.  We  simply  cannot  afford  what  Dr.  Fubini  has  termed  the  American 
syndrome,  the  penchant  for  complex  gadgetry. 

Both  complexity  and  gadgetry  provide  additional  opportunity  for 
failures  to  occur.  These  failures  can  be  machine  failures  or  they 
can  be  man  failures  stemming  from  the  sheet  complexity  of  his  work 


environment.  As  a  result  today's  sophisticated  systems  show  a  tendency 
toward  undependability.  They  are  marvels  of  tecnnological  ingenuity. 

But  —  this  is  little  comfort  to  the  Commander  unable  to  complete  his 
mission  because  the  beautifil  beast  cannot  be  relied  upon  to  work 
when  and  where  needed. 

Undependability  stems  from  many  contributing  factors  such  as 
shortcomings  in  reliability,  maintainability,  compatibility i  human 
factors,  logistics  supportability,  etc.  All  of  these  qualities  of 
systems  are  fairly  readily  understood  and  quite  well  appreciated  both 
by  military  and  industry.  But  —  understanding  and  appreciation  are 
not  enough.  We  must  devise  valid  means  for  measuring  these  qualities. 
This  is  necessary  so  that  we  can  assure  dependability  by  effecting  real 
communication  between  the  military  and  industry  in  specifying  mutually 
acceptable  and  attainable  contract  requirements,  and  in  insuring  that 
these  specified  qualities  of  dependability  are  in  fact  achieved  by 
the  contractor. 

It  would  appear  that  in  our  haste  to  keep  up  with  technology,  we 
have  not  expended  the  time  or  effort  needed  to  consolidate  and  stabilize 
the  technology  we  have  exploited.  This  is  not  tc  say  thst  the  problem 
has  been  ignored.  On  the  contrary  far  more  attention  has  been  given  to 
system  effectiveness  than  can  be  readily  measured.  Each  contributing 
designer  in  a  system  is  conscious  of  the  need.  In  his  own  area  he 
designs  to  the  end  of  realizing  dependability.  This  is  not  enough. 

We  must  insure  that  all  of  the  foregoing  factors  are  mutually 
optimized  on  a  total  system  basis.  This  must  be  done  before  we  provide 
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In  doing  this,  ve  oust  recognize  and  fully  face  a  basic  anomaly  of 
the  situation.  Technology  is  a  dynamic  force*  How  then  can  it  be 
consolidated? 

I  would  suggest  that  it  can  be  consolidated  in  the  same  manner 
in  which  it  can  be  measured — at  a  discrete  point  in  time.  We  must  pick 
this  point  end  stay  with  it.  Then  both  the  military  and  industry  must 
resist  the  temptation  to  inject  incremental  advances  in  the  course  of 
a  development.  Ve  must  defer  these  goodies  to  a  subsequent  generation. 
There  are  those  who  will  take  issue  with  this  position.  They  say  that 
this  assures  obsolesence  upon  delivery.  I  submit  that  this  argument 
is  more  hypothetical  than  real.  Nevertheless,  even  given  that  there 
is  some  validity  to  the  argument,  I  would  submit  that  our  military 
posture  is  strongest  when  systems  have  mission  dependability.  This 
holds  even  without  the  latest  theoretical  increment  of  capability. 

For  instance,  a  destroyer  with  a  sonar  system  reliability  of  .9  and 
a  probability  of  detection  of  .8  at  a  maximum  range  of  10,000  yds  is 
of  more  use  to  the  ASW  Commander  than  one  with  a  .5  reliability  and  a 
12,000  yd  max  range  even  assuming  comparable  probabilities  of  detection. 
Consider  also  that  lov  reliability  can  normally  be  expected  to  be 
accompanied  by  reductions  in  the  other  qualitative  factors  that  I’ve 
cited.  Recognizing  the  validity  of  this  type  of  reasoning,  the  Navy 
is  taking  a  hard  look  at  its  RJH)  Program. 

We  in  the  Office  of  Naval  Material  call  this  examination  System 
Effectiveness  analysis.  Ve  are  studying  the  trade  offs  between 
technological  advances  and  consolidation  of  technological  position. 


r 


As  we  learn  more  in  the  skills  of  quantizing  the  factors  which  to  go 
sake  up  Systems  Effectiveness!  the  examination  will  be  even  more 
rigorous.  This  nay  well  result  in  a  reduction  of  the  number  of  systems 
under  development  but  with  a  more  intensive  effort  on  those  remaining. 

Ve  -are  persuaded  that  the  key  to  development  pay-off  is  the 
successful  consolidation  of  position  through  a  viable  approach  to 
Systems  Effectiveness.  The  analysis  and  review  by  the  Office  of  the 
Secretary  of  Defense  under  Project  Definition  Phase  and  under  Confi¬ 
guration  Control  require  some  such  approach.  Further  we  need  to  pursue 
such  a  course  to  permit  real  calculation  of  risk.  Since  last  December 
2nd,  all  Proposed  Techlncal  Approaches,  Technical  Development  Plans, 
Specific  Operational  Requirements  and  Advanced  Development  Plans  have 
received  rather  penetrating  review  by  my  Systems  Effectiveness  Group. 

This  review  will  be  intensified  in  depth  with  a  view  toward  recommending 
the  elimination  of  incremental  improvement  effort  that  is  not  completely 
justified  by  urgent  military  worth  considerations.  Our  goal  is  to  in¬ 
sure  that  all  factors  of  Systems  Effectiveness  receive  thorough  attention 
and  adequate  consideration  before  the  production  phase  and  fleet  intro¬ 
duction. 


This  may  sound  like  a  hardnosed  approach.  It  is  not  intended  to 
be.  However,  it  is  intended  to  be  a  hard-headed  approach.  There  simply 
is  no  substitute  for  effectiveness  in  a  system  -  for  -  if  the  system 
does  not  have  mission  effectiveness,  where  is  the  pay-off?  The  pay-off 
comes  only  on  effective  results.  So,  we  in  the  Office  of  Naval  Material 
intend  to  take  a  page  out  of  the  book  of  some  of  the  good  people  of 
Montana.  They  tell  me  that  no  matter  how  well  configured  he  may  be, 
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the  sterile  ram  gets  shot  and,  like  ewes,  rapidly  become  mutton.  I 
don’t  know  what  boiled  missile  tastes  like  or  roast  radar  —  but  I’m 
willing  to  find  out. 
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In  recent  years  many  cults  have  arisen  each  of  which  has  indicated 
a  real  need  for  management  actici  to  cet  system  effectiveness.  The 
time  has.  come  to  relate  all  these  cults  to  each  other  and  to  relate 
them  to  the  basic  management  objective.  The  1st  v.igraph  "Integration 
of  Assurance  Cults"  identifies  some  of  the  major  product  characteristic 
cults  and  some  of  the  major  program  management  cults.  It  states  that 
both  types  are  only  subordinate  factors  in  achieving  the  basic  objective 
of  svstem  effectiveness  and  consequently  they  should  be  treated  as  vital 
but  subordinate  segments  of  an  integrated  system  effectiveness  assurance 
management  system. 

No  one  questions  the  objective  of  system  effectiveness  assurance. 
Many  people  do  question  the  need  for  a  formalized  management  system 
to  achieve  these-objectives.  This  resistance  must  be  faced  squarely  and 
understood.  The  2nd  vugraph  "Yearning  for  Anarchy*  summarizes  the 
opposition  to  any  management  system  that  constrains  the  activities  of 
people.  It  raises  the  vital  question  "Why  should  we  annoy  everybody 
with  a  management  system?"  Since  the  purpose  of  management  is  to  get 
things  done  through  people  we  must  have  a  good  reason  for  doing  any¬ 
thing  contrary  to  the  nature  of  people. 

The  3rd  vugraph  "Justification  for  Discipline"  states  two  major 
reasons  for  a  formalized  management  system.  It  conceives  that  the 
system  does  seek  to  control  the  activities  of  people  but  only  to  the 
extent  necessary  to  assure  the  two  major  objectives  of  requirements 
optimization  and  experience  retention.  Requirements  optimization  in- 
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sties  that  are  critical  to  these  objectives. 
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volves  techniques  of'  system  analysis.  Cdr  Sargent  has  published  some 
excellent  papers  on  the  subject  and  it  will  be  dealt  with  in  more 
detail  at  the  second  symposium  on  Nov  17. 

In  regard  to  experience  retention  every  project  shows  that  in 
fact  we  do  repeat  mistakes  and  do  so  over  long  periods  of  time.  For 
example,  at  a  recent  3IMRAB  meeting  a  presentation  on  aircraft  safety 
summariaad the  causes  of  loss  of  property  and  life  in  a  modern  .jet 
aircraft.  Then  the  speaker  showed  a  picture  of  a  30-year  old  biplane 
and  made  the  key  point.  The  point  was  that  the  same  cause  of  failure 
for  the  modern  jet  that  had  occurred,  had  been  analyzed  and  understood 
30  years  ago. 

Requirements  optimization  is  achieved  by  disciplined  decision  making 
based  on  system  analysis  techniques.  Experience  retention  is  merely 
the  application  of  the  discipline  of  the  scientific  method  to  management 
problems. 

The  4th  vugraph  "Definition  of  Discipline"  identifies  this  word 
with  the  training  motivation  commanding  and  auditing  of  people.  The 
management  system  does  .not  seek  to  provide  even  these  types  of  formal 
discipline  except  for  those  activities  that  experience  has  shown  to  be 
critical  to  achieving  system  effectiveness,  .nlso  this  vugraph  summarizes 
the  three  vital  steps  in  the  management  system.  Briefly  they  are: 

(1)  Critical  activity  iaentif ication 

(2)  Resources  development 

(3)  Resources  application  for  each  critical  activity 

This  description  corresponds  with  the  definition  that  "The 'business  of 


FORMAL  RESOURCES  APPLICATION  THROUGH  PROGRAM  MANAGEMENT  TECHNOLOGY 


management  is  resources  development  and  resources  application  to  acnieve 
pre-determined  objectives". 

The  5th  vugraph  "Recognition  of  Need"  quotes  fir.  McNamara  on  disciplined 
decision-making  to  assure  requirements  optimization.  Also  it  quotes 
Gen  Schriever  on  discipline  execution  of  the  program  management  function 
after  a  project  has  been  authorized.  Quotations  from  Navy  sources  have 
been  omitted  because  they  are  appropriate  to  the  next  meeting  which 
deals  with  the  system  effectiveness  activities  of  the  Office  of  Naval 
Material. 


SYSTEM  EFFECTIVENESS  GRTI 


ACTIVITIES 


The  6th  vugraph  "Decision  and  Hardware  SECA"  stresses  that  the 


management  system  is  concerned  with  the  activities  of  people.  It  defines 
what  is  meant  by  a  system  effectiveness  critical  activity  and  illustrates 
the  relationship  of  two  types  of  activity  to  their  products. 

It  should  be  noted  that  although  hardware  is  the  primary  product, 
three  types  of  data  also  are  products. 

The  7th  vugraph  "Design  Critical  Activities"  illustrates  four 
types  of  SECA  and  their  relationship  through  design  decisions.  A 
management  system  does  not  seek  to  discipline  the  instantaneous  working 
of  a  man's  brain  when  he  is  using  his  judgment  to  make  a  decision. 

Rather  it  seeks  to  assure  that  appropriate  and  reliable  data  is  genera¬ 
ted  and  is  used  as  a  basis  for  decision  making. 

The  8th  vugraph  "Types  of  SECA"  illustrates  the  first  step  in 
cataloging  both  decision  or  hardware  or  work  SECA.  A  review  and  analysis 
of  reliability  specifications  can  lead  to  identification  of  reliability 
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SECA.  Similarly  a  review  and  analysis  of  maintainability  specifications 
can  lead  to  identifying  activities  that  the  writer  considered  to  be 
critical  to  system  effectiveness. 

SYSTEM  EFFECTIVENESS  ASSURANCE  MANAGEMENT  SYSTEM 

So  far  we  have  defined  only  creation  activities.  This  means 
activities  that  actually  create  or  build  in  system  effectiveness  or 
some  worn  characteristic  such  as  reliability  that  contributes  to  system 
effectiveness.  It  is  entirely  practical  to  produce  a  product  such  as 
a  lawnmower  by  nothing  but  creating  activities.  Yet  we  know  that  for 
military  programs  we  do  require  and  perform  other  activities  such  as 
qualification,  testing  and  receiving  inspection.  These  activities  do 
not  create  but  they  help  assure  system  effectiveness. 

The  Navy  has  recognized  the  importance  of  assurance  activities 
in  many  ways.  For  example,  years  ago  the  then  Chief  of  the  Bureau  of 
Naval  Ordnance  required  the  Commanding  Officer  of  the  Naval  Ordnance 
Laboratory  to  establish  and  operate  an  evaluation  for  assurance  function 
independent  of  the  development  or  creation  function.  Why  does  he  bother 
with  these  nonrcreative  assurance  activities? 

The  9th  vugraph  ’’Purposes  of  Assurance"  seeks  to  answer  this 
question.  Several  types  of  assurance  activity  such  as  qualification 
testing  are  like  a  hurdle  that  the  program  manager,  designer,  or  machinist 
must  overcome.  The  knowledge  that  he  must  do  so  makes  it  more  probable 
that  he  will  do  a  thorough  job  of  performing  system  effectiveness  by 
creating  activities.  The  importance  of  increasing  confidence  that 
effectiveness  has  been  built  into  a  system  is  well  illustrated  by  the 
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Apollo  Man  Spaqs  Program.  Certainly  the  NASA  official  responsible  for 
the  decision  to  launch  man  into  space  must  be  able  to  assure  the  public 
that  every  possible  means  of  checking  that  a  safe,  reliable  system  has 
been  created  is  used. 

The  10th  vugraph  "Relation  of  Assurance  Activities  to  SECA"  is  the 
most  vital  of  all. 

The  most  familiar  types  of  assurance  are  those  applied  after  the 
product  has  been  completed.  They  include,  design  reviews,  qualifica¬ 
tion  testing,  and  receiving  inspection.  But  after  the  fact  assurance 
is  not  enough.  We  must  assure  that  resources  are  developed  for  each 
critical  activity.  This  is  done  through  the  first  three  types  of 
assurance.  Then  we  must  assure  that  for  each  project  the  critical 
activities  are  required,  funded,  and  scheduled.  This  is  done  through 
program  planning.  Also  we  must  assure  that  these  critical  activities 
are  being  performed  well  curing  the  time  they  are  being  performed.  This 
is  done  through  important  surveillance. 

The  six  types  of  assurance  together  with  the  identification  of  critical 
activities  constitute  the  system  effectiveness  assurance  management 
system.  The  nature  of  this  system  may  be  illustrated  many  ways. 

Vugraph  11  "Resources  Development  and  Application"  relates  the 
system  to  the  definition  of  management  as  resources,  development  and 
application  to  achieve  pre-deterrained  objectives. 

The  12th  vugraph  "Dynamic  Closed  Loop"  illustrates  that  the  system 


works  continuously  by  feedback  and  integration. 

The  13th  vugraph  "Data  Acquisition  Foundation"  illustrates  the 
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that  the  whole  system  is  based  on  acquiring  data  from  both  a  project 
data  system  and  functional  audit  system. 


The  14th  vugraph  "Audit  Checklist”  is  important  because  it  provides 
a  basis  for  checking  the  status  of  current  operations.  For  example  it 
would  permit  ONM  to  check  whether  for  a  critical  activity -such  as  SOR 
writing  data  was  being  required  on  the  successes  and  failures  of  present 
SORs.  It  would  allow  checking  whether  research  money  was  being  used  to 
develop  and  document  techniques  for  writing  satisfactory  SORs  and  it 
would  permit  checking  whether  the  training  and  motivation  of  officer 
personnel  included  adequate  instruction  in  the  writing  of  SORs. 

The  15th  vugraph  "Emphasis  on  People”  merely  reminds  us  that  we  are 
not  dealing  with  an  academic  system  but  a  down  to  earth  practical  method 
of  achieving  results  through  people.  in. fact  people  are  the  link  between 
resources,  development  by  functional  executives  and  resources  application 
by  program  managers. 

The  16th  vugraph  summarizes  the  segments  of  the  assurance 
management  system  (also  it  stresses  the  acronyms  seca,  seer,  pmt,  and 
seams) 

SE  EXPERIENCE  RETENTION 

So  much  for  the  overall  management  system.  The  system  takes  on 
much  more  meaning  when  we  get  into  the  mechanics  of  resources  develop¬ 
ment  through  experience  retention  and  resources  application  through 


program  management  technology.  We  will  discuss  these  topics  briefly 
then  get  down  to  operating  problems  with  a  discussion  of  "contracting 
for  system  effectiveness". 


SYSTEM  EFFECTIVENESS  ASSURANCE  MANAGEMENT  SYSTEM 


engineering 


SYSTEM  EFFECTIVENESS  ASSURAN 


J 


•To  be  replicable  the  development  of  system  effectiveness  resources 
must  comply  with  the  scientific  method.  The  17th  vugraph" deals  with  the 
scientific  method*  It  shows  that  the  6  segments  of  our  assurance 
management  system  correspondence  quite  precisely  with  the  recognized 
four  steps  of  the  scientific  method.  It  is  important  to  note  that  the 
analysis  step  does  not  correspond  with  a  physical  description  of  a 
mode  of  failure.  We  as  managers  cannot  prevent  failures  by  changing 
the  laws  of  physics  but  only  by  controlling  the  activities  of  people. 
Consequently,  our  analysis  is  concerned  with  how  successes  can  be 
repeated  or  failures  prevented  from  oceuring  through  controlling  the 
activities  of  people.  Simarly  our  hypothesis  step  consists  of  predicting 
certain  changes  in  the  training, motivation,  direction  or  audit  of  people 
will  cause  them  to  behave  in  a  desired jmanner. 

The  18th  vugraph  illustrates  the  principles  of  orgaiization  for 
experience  retention.  A  functional  executive  for  reliability  or  for 
system  effectiveness  does  not  need  a  line  organization  if  he  has  the 
authority  to  make  a  closed  loop  program  work.  Irrespective  of  the 
organizational  position  of  the  people  who  perform  each  of  the  6  segments 
of  the  closed  loop  system  certain  things  must  be  provided  for.  Experience 
retention  engineers  preferably  in  a  data  central  organization  must  generat 
(l)  failure  mode,  probability  and  effects  data,  (2)  a  catalog  of  activitie 
requiring  formal  discipline  and  (3)  disciplinary  requirements  for  each 
such  activity.  These  requirements  are  passed  on  to  the  appropriate 
functional  executive.  He  in  turn  absorbs  the  lessons  learned  into  his 
management  system  in  any  way  he  chooses  providing  it  will  be  effective. 
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_ _ PROJECT  EXECUTIVES _ 

OLD  PROJECT  |  NEW  PROJECT 

EXPERIENCE  — SUCCESS  AND  FAILURE  CASE  HISTORIES 


Each  functional  executive  produces  resources  in  the  form  of  documented 
disciplines  and  qualified  people.  The  requirement  for  using  these  re¬ 
sources  on  each  new  project  completes  the  closed  loop  system.  The  title 
of  the  19th  vugraph  is  "Discipline  Requirements  Yes  -  Procedures,  No". 

Again  w>.  "^e  facing  the  facts  of  human  nature  and  seeking  to  answer  two 
types  of  very  real  opposition  to  formalised  experience  retention. 

The  20th  vugraph  "System  Effectiveness  Activities  and  Experience 
Retention"  summarizes  the  responsibilities  of  functional  executives. 

PROGRAM  MANAGEMENT  TECHNOLOGY 

The  21st  vugraph  "What  is  the  System  Effectiveness  Activity*'  ex¬ 
presses  the  point  of  view  of  the  program  manager.  From  his  point  of 
view  a  SECA  is  so  vital  that  it  must  be  assured  by  planning  surveillance 
and  evaluation. 

The  22nd  vugraph  brings  out  the  importance  of  documentation  to  an 
assurance  management  system.  Each  of  the  three  types  of  program  manage¬ 
ment  assurance  involve  documents.  Program  Planning  involves  documented 
plans.  Surveillance  is  based  on  in  process  data.  Evaluation  includes 
evaluation  of  decision  disclosure  documents  and  the  documentation  of 
test  and  operational  results. 

For  any  assurance  system  to  be  accepted  wholeheartedly  and  implemented 
people  must  have  a  positive  attitude  to  the  written  word.  A  cavalier  atti¬ 
tude  that  paper  work  may  be  justified  because  of  the  existence  of  a  great 
deal  of  bad  paper  work  is  fatal  to  system  effectiveness  assurance. 

The  23rd  vugraph  is  a  quotation  from  Gen  Schriver.  It  forcibly  expresses 
his  opinion  developed  while  responsible  for  the  highly  successful  ICBM 
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EXPERIENCE  MUST  BE  ANALYZED  AND  DISCIPLINE  REQUIREMENTS 
EXPRESSED  IN  A  WAY  THAT  MAKES  THEM  APPLICABLE  TO  NEW  PROJECTS. 


VUGRAPH  20 
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NECESSITIES 


VUGRAPH  22 


VUGRAPH  23 


,.ie  24th  vugraph  illustrates  the  tough  action  that  is  being  taken 
by  the  Mr  Force  to  insist  on  discipline  program  management  by  their 
contractors . 

The  25th  vugraph  illustrates  the  action  being  taken  by  the  Army 
Material  Command  to  assure  discipline  decision  making  in  their  program 
management. 

The  26th  vugraph  quotes  from  the  DOD  news  release  when  directive 
3200.9  was  issued  on  March  4,  1964.  It  illustrates  DOD  In  requirenenta 
on  disciplined  program  management  action  at  the  beginning  of  each 
program. 

Again  we  have  avoided  quotations  from  Navy  sources  because  this 
is  part  of  the  subject  of  the  next  seminar. 

OUTPUT  CONTRACTING  FOR  &STBMS  EmCTTVENBSS 

The  traditional  way  of  doing  business  may  be  described  by  the  term 
output  contracting.  This  means  thit  the  relationship  between  the  buyer 
and  the  seller  is  based  exclusively  on  requirements  and  tests  of  the 
final  product. 

This  is  the  third  record  requirements  for  and  test  of  the  final 
product.  Vugraph  27  "Necessary  Conditicna",summarize  jutput  contracting 

Even  when  the  relationship  between  the  buyer  and  seller  is  based 
exclusively  on  output  requirements  it  is  still  necessary  to  optimize 
these  requirements  through  systems  analysis.  Also  the  overall  system 
requirement  must  be  converted  into  contributing  characteristic  require¬ 
ments.  These  characteristics  have  to  be  things  that  can  be  required 
and  specified  at  the  appropriate  level  of  contracting  including  pur- 


SURPRISED  IN  A  COMPETITIVE  ENVIRONMENT. 


PROGRAM  MANAGEMANT  TECHNOLO 


VUGRAPH  25 


MAJOR  BENEFITS  ARE:  REDUCTION  OF  TECHNICAL  CHANGES,  DECREASED  TOTAL 
COST  IMPROVED  OPERATIONAL  EFFECTIVENESS,  EARLY  CANCELLATION. 
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AND  RECEIVING  INSPECTION. 


chase  orders  for  parts.  Vugraph  28  “Requirements  Optimization"  illustrates 
thas~  points. 

Vugraph  29  "Derivation  of  Requirement"  illustrates  three  steps 
from  overall  operational  effectiveness  concepts  to  quantitative 
requirements  suitable  for  inclusion  in  contracts  and  purchase  orders. 

The  technology  for  making  trade-offs  that  result  in  contractor  require¬ 
ments  is  a  very  large  subject  which  we  will  not  be  able  to  discuss 
today.  The  important  point  is  that  these  quantitative  analyses  and 
trade-offs  are  essential  to  supporting  output  contracting  for  Systems 
Effectiveness. 

INPUT  CONTRACTING  FOR  SYSTEMS  EFFECTIVENESS 

When  a  buyer  cannot  afford  the  risks  associated  with  output 
contracting  he  must  take  action  to  assure  that  system  effectiveness  is 
built  in  from  the  beginning  of  the  program.  We  call  such  action  input 
contract. 

There  have  been  many  attempts  to  require  input  contracting  to 
assure  a  single  characteristic  such  as  reliability  or  maintainability. 
Vugraph  30  "Specifications  That  Require  Discipline  Procedures"  illustrates 
the  attempts  to  establish  input  con''  iting  through  specifications. 

There  has  been  a  great  deal  lustry  persistence  to  specifica¬ 

tions  that  seem  to  direct  management  practices  in  the  name  of  a  single 
cult.  It  is  being  realized  slowly  that  government- industry  cooperation  in 
formalized  overall  program  management  practices  can  eliminate  the  need 
for  the  management  aspects  of  a  large  number  of  specifications  such 
as  MIL-Q  9853A. 

Even  when  industries  has  accepted  principle  of  input  contracting 
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INPUT  CONTRACTING 


the  objective  is  still  to  limit  the  amount  of  constraint  to  the 
minimum.  Vugraph  31  "Industry  Objectives”  clarifies  the  relation  of 
this  attitude  to  Systems  Effectiveness  critical  activities. 

Vugraph  32  "Steps  in  Constraint  of  Contractors”  illustrates  that 
industry  wishes  to  use  that  method  of  contracting  which  requires  the 
minimum  degree  of  constraint  for  a  particular  procurement.  Whenever 
possible  both  buyer  and  seller  still  prefer  to  do  business  on  the 
basis  of  pure  output  contract.  At  the  other  extreme  large  new  system^ 
that  involve  a  group  of  associated  contractors  and  many  innovations 
in  technology  require  the  illustrated  additional  disciplines  in 
program  planning  assurance,  inputs  surveillance  assurance,  and  output 
evaluation  assurance. 

From  the  industry  point  of  view  a  formal  system  effectiveness 
assurance  management  system  makes  sense  if,  l)  All  the  critics! 
activities  that  are  called  out  in  the  contract  are  well  defined  and 
have  been  proved  to  be  critical  by  previous  experience.  2)  The  customer 
encourages  and  rewar  as  resourcefulness  of  development  by  industry  for  each 
critical  activity.  3)  The  customer  has  the  qualified  people 
and  proven  methods  necessary  to  evaluate  industry's  effort  in  applying 
resources  to  each  critical  activity. 

Unfortunate!'' .  a  great  deal  remains  to  be  done  by  both  government 
and  industry  before  these  conditions  can  be  considered  satisfactorily 
fulfilled.  What  has  been  done  by  specifications  leaves  much  to  be 
desired.  For  example,  the  truly  critical  activities  are  not  well 
defined  nor  is  their  criticality  established  by  straightforward  govern- 
raen+ -industry  evaluation  and  agreement. 
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The  time  has  come  for  mutual  industry-government  action  in  describ¬ 
ing  and  implementing  a  totally  integrated,  fully  justified  system 
effectiveness  assurance  management  system.  Such  a  system  will  help 
immeasurably  not  only  to  get  effective  systems  for  the  Navy,  but  to 
assure  harmonious  relations  between  the  Navy  and  Navy  contractors. 

Mutual  confidence  and  respect  is  an  important  by-product  that  should 
be  achieved  by  any  system  effectiveness  assurance  management  system. 
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Dr.  Trumbull,  ladies  and  gentlemen 

The  term  "System  Effectiveness"  in  a  generic  sense  is  usually 
understood  to  mean  the  ability  of  a  system  to  perform  according  to 
its  purpose.  Since  this  is  a  qualitative  characteristic,  it  is 
difficult  at  the  outset  to  envision  it  as  a  tool.  However,  if  we 
were  able  to  quantize  the  various  attributes  of  a  system  which  go 
to  make  up  this  quality  we  could  establish  measures  of  the  quality. 
Having  established  measures,  these  can  be  compared  in  a  number  of 
ways:  predicted  vs  achieved,  achieved  at  one  point  in  time  vs  that 
of  a  later  point  in  time,  predicted  or  achieved  in  one  system  vs 
predicted  or  achieved  of  another,  one  combination  of  attributes  vs 
a  differing  combination  of  attributes,  among  others.  These  ccmpari- 
sions  are  the  vehicles  for  appraisals  which  become  the  tools  oi 
management  in  making  the  decisions  which  determine  the  future  course 
of  our  efforts. 

This  is  a  nice  rationale  but  it  is  conditional  upon  the  big 
"if"  at  the  outset.  —  If  we  are  able  to  quantize.  Actually  I  am 
persuaded  that  "if"  may  not  be  the  right  word.  Rather,  I  would 
suggest  that  we  should  use  the  word  "when".  This  stems  from  the 
conviction  that  the  condition  is  not  a  matter  of  whether  or  not 
quantizing  is  possible  but  rather  one  of  whether  or  not  we  are 
capable  of  learning  how  to  quantize. 
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In  support  of  this  contention,  let  us  examine  one  proposed 
approach  to  quantifying  systems  effectiveness.  This  approach  was 


f  » 

presented  by  the  Cost  Effectiveness  Panel  at  the  recent  Aerospace 
Reliability  and  Maintainability  Symposium  in  Washington  and  is  pub¬ 
lished  in  its  Proceedings. 

Let  me  first  present  *he  generalized  formula  and  then  pursue 
its  derivation.  Actually  two  formulae  were  presented. 

(1)  Ec  =  PAU  Ec  =  W  PAU 

Ca  +Cu  ca  + 

Although  there  is  a  significant  difference  between  the  two  in 
that  the  W  is  a  constant  representing  the  military  worth  of  the 
mission  of  the  system,  I  shall  for  the  moment  use  the  first  of  the 
two  formulae.  The  term  Ec  represents  Cost  Effectiveness.  I  would 
remind  you  that  we  are  not  discussing  Cost  Effectiveness  per  se,  but 
rather  Systems  Effectiveness.  Nevertheless  Systems  Effectiveness 
taken  out  of  the  context  of  Cost  Effectiveness  is  a  sterile  academic 
exercise  productive  of  little  but  to  impress  our  fellows  with  our 
erudition.  Since  we  are  pursuing  a  useable  tool,  I  have  elected  to 
keep  Systems  Effectiveness  in  a  useful  context  ie:  within  Cost 
Effectiveness . 

Looking  to  the  denominator  of  the  fraction  in  (l)  we  find  the 
terms  Ca  &  Cy.  These  are  Cost  of  Acquisition  and  Cost  of  Utilization 
respectively.  These  are  the  terms  which  convert  the  expression  to  a 
statement  of  Cost  Effectiveness.  These  are  the  determinants  of  the 
cost  context.  I  shall  not  pursue  their  derivation  in  this  particular 
discourse. 
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There  remain  three  terms,  P,  A  &  U.  These  provide  the  substance 


of  our  systems  effectiveness  model,  where 
(2)  E  =  PAU 

The  term  P  -  Performance  is  a  numerical  index  expressing  system 
capability,  assuming  a  hypothetical  IOC#  availability,  reliability 
and  utilization  of  performance  capability  in  actual  operation.  This 
index  can  be  expressed  in  any  suitable  terms  dependent  upon  the  nature 
of  the  mission.  For  instance  in  a  satellite  system  it  could  be  in 
rated  pounds  of  payload  into  orbit  per  successful  vehicle.  In  a 
missile  system  it  could  represent  target  square  miles  killed  per 
successful  launch.  In  effect  it  is  the  mission  determinant  parameter. 
It  matters  little  what  the  index  selected  may  be,  provided  that  in  the 
use  of  the  formula,  the  index  used  is  the  determinant  parameter  for  the 
systems  being  appraised  through  comparison  ie:  in  the  determination 
of  Pq,,  P^,  By  etc.  where  these  values  obtain  for  systems  or,  g,  y 
respectively. 

The  term  U,  Utilization  is  the  fraction  of  the  performance  capa¬ 
bility  actually  utilized  due  to  the  specific  application  and  the 
environment  encountered.  In  effect  it  expresses  all  of  the  effective¬ 
ness  degradation  due  to  causes  external  to  the  system  itself.  I 
would  point  out  that  this  is  a  term  largely  out  of  the  control  of  the 
designer,  and  one  which  in  the  final  analysis  can  only  be  established 
ex  post  facto.  Nevertheless  it  is  a  function  which  can  vary  as  a 
result  of  inherent  design  limitations.  Such  factors  as  consideration 
of  operational,  physical  and  in  some  cases  political  environmental 


factors  operate  on  this  function.  These  considerations  are  under 
the  designer's  control  albeit  subject  to  the  adequacy  of  communication 
of  these  environmental  parameters  by  the  user.  Because  of  this 
designer  control,  the  value  U  must  be  considered  if  only  with  a 
probabilistic  estimated  value.  However,  the  analyst  MUST  exercise 
great  objectivity  and  care  to  insure  that  a  biasing  effect  is  not 
introduced  through  failure  to  develop  Ua,  Up  &  Uy  on  equitable  bases. 

The  central  term  A  is  the  period,  or  fraction  of  time,  that  the 
system  is  ready  and  capable  of  fully  performing  its  mission.  It  makes 
little  difference  whether  A  is  expressed  as  a  period  or  a  fraction  as 
long  as  the  mode  of  expression  is  consistent  throughout  a  given 
appraisal  or  analysis.  In  those  systems  having  a  finite  mission  time, 
it  is  convenient  to  use  the  fractional  expression. 

Conventionally  A  is  expressed  as 

(2)  A  »  MTHF 

MTBP  +  MTTR 

I  would  point  out  that  this  condition  exists  only  at  the  point 
t  =  0.  It  is  comforting  to  know  that  a  system  is  ready  to  go.  But 
in  a  military  system,  the  payoff  is  in  successful  completion  of  the 
mission,  t  =  0  or  t0  is  Just  the  beginning.  We  must  have  an  expression 
which  will  permit  us  to  predict  A  at  the  point  of  tw,  the  completion  of 
the  mission. 

To  the  end  of  realizing  such  an  expression  of  A,  I'd  like  to 


state  some  defintions  of  terms  followed  oy  the  derivation  of  an 
expression  of  A  which  appears  to  be  useful.  (See  Fig  #l) 


(T)  Stress  Time  is  the  time  during  which  stresses  occur  that 
can  induce  failure.  Such  stresses  are  normally  present  during  some 
fraction  or  all  of  "operational"  time,  and  also  usually  during 
standby  and  maintenance  time.  This  chart  illustrates  the  major 
elements  of  stress  time,  but  as  a  ratio  to  the  number  of  failures: 


All  times 
are  averages 


MTBF  (Stress  time  per  failure)  T 


(t)  Mission  Time  is  the  time  during  which  the  system  is 
committed  to  completion  of  one  operational  mission. 

Mean  Time  Between  Failures  is  the  average  stress  time 
between  failures,  assuming  no  redundant  compensation  unless  specified. 
It  is  the  primary  index  of  design  reliability,  and  equal  to  the 
reciprocal  of  Failure  Hate  1. 

(R)  Reliability  is  the  fraction  of  successful  mission 
starts  which  have  been  or  will  be  subsequently  completed  without 
failure.  Predicted  Reliability  Rp  is  then  the  probability  that, 
successfully  started,  the  system  will  complete  operation  for  a 
specified  time  without  failure,  or 

(4)  Predicted  Reliability  Rp  =  e  “^/T  w  q  _  t/T  when  t  «  2 
Downtime  Per  Failure  is  the  mean  time  for  restoration  (MTTR)  to 
reliable  operation,  including  detection,  diagnosis,  logistic  procure¬ 
ment,  replacement  or  repair,  checkout,  and  (for  memories)  reload. 
"Repair"  time  is  but  part  of  this. 

(M)  Maintainability  is  the  quantitative  index  of  ease  with 
which  restoration  is  accomplished.  Typical  measures  are  (a)  the 
number  of  failures  restored  per  hour  of  downtime  (M  =  l/D),  (b)  the 
fraction  of  attempts  wherein  restoration  is  completed  in  a  specified 
time,  and  (c)  operational  time  per  dollar  cost  of  preventive  and 
corrective  maintenance. 

(A(j)  Demand  Availability  is  the  fraction  of  stress  time 
that  upon  demand  the  system  can  be  operated  without  failure,  assuming 
that  any  preventive  maintenance  can  be  interrupted.  It  is  equal  to 
the  probability  that  the  system  can  start  successfully  and  complete 


the  required  mission  time  without  failure,  at  any  random  demand  time. 
Since  the  mission  will  fail  if  it  starts  any  later,  on  the  average, 
than  time  r.  before  failure,  or  within  downtime  D  following  failure. 


(5)  Ad  =  1 


fc+D 


T-tps  +•  D-Ds 

where  Ds  is  that  portion  of  downtime  D  during  which  stress  occurs, 
and  tpS  is  that  portion  of  preventive  maintenance  time  tp  during 
which  stress  occurs. 

If  there  is  stress  during  substantially  all  of  downtime,  which 
is  usually  the  case,  Ds  =  D.  Then  if  there  is  no  stress  during 
preventive  maintenance,  tps  =  0  and  we  have 

(6)  Ajj  » 1  -  t  +  D 

T 

If  there  is  no  stress  during  downtime,  which  is  rare,  Ds  =  0. 


Then  if  there  is  still  no  preventive  stress,  tpS  =  0  and  : 

(7)  Ad  »  1  -  t  +  D  =  T  -  t 

T  +  D  T  +  D 

If  we  ignore  the  reliability  effect  of  incomplete  missions  on 
Availability,  t  =  0.  Then  we  have  the  familiar  form: 

(8)  Ad  »  T  =  MCHF 

T  +  D  MTHF  +  MTTR 

(Ac)  Continuous  Availability  is  the  fraction  of  stress  time 
that  the  system  can  be  operated  without  failure,  allowing  for  both 
downtime  and  preventive  maintenance. 

(9)  A  =  1  -  t  +  D  +  tp  =  1  -  t  »  P  +  tn 

T-tpS  +  D-Ds  i 

(U)  Utilization  is  the  fraction  of  performance  capability 
actually  utilized  due  to  the  specific  application  and  environment 
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encountered.  It  expresses  all  effectiveness  degradation  due  to  causes 
external  to  the  system  itself. 

Example:  63$  Tactical  Effectiveness  due  to  narrow  applica¬ 
tion  (70$)  and  adverse  environment  (90$) 

It  is  this  Continuous  Availability  expression  which  provides  a 
value  for  A  which  will  permix  us  to  predict  out  to  the  point  in  real 
time  t®.  Thus  our  expression  can  be  written 

(10)  E  =  P  (l  -  t  +  D)  U 
T 

At  this  juncture,  '  Atld  like  to  acknowledge  my  indebtedness  to 
the  Messrs.  E.  S.  Winlund  and  C.  S.  Thomas  for  their  contributions  to 
this  derivation. 

There  is  one  remaining  term  in  our  expression,  E.  I  have  previously 
indentified  this  term  as  Effectiveness  or  more  precisely  Systems 
Effectiveness.  Further,  I  expressed  it  as  a  generic  term.  To  avoid 
misunderstanding  I  would  provide  a  more  exact  definition  of  the  term 
in  the  context  which  it  is  used  here.  The  conceptual  definition  is 
the  probability  that  the  system  can  successfully  meet  an  operational 
demand  throughout  a  given  time  period  when  operated  under  specified 
conditions.  I’m  indebted  to  WSEIAC  Group  I  (AFSC’s  Weapons  System 
Effectiveness  Industrial  Advisory  Committee)  for  this  phrasing.  The 
mathematical  definition  must  be  the  index  of  the  conceptual  definition 
since  the  analyst  may  elect  to  use  a  non-fractional  expression  of  the 
term  U  and,  the  term  P  is  usually  neither  fractional  nor  the  percentile 
expression  normally  associated  with  probabilities. 

Having  arrived  at  a  generalized  expression  of  Systems  Effectiveness, 
let  us  look  at  the  utility  of  the  tool.  A  prudent  mechanic  knows  not 


use. 

Further,  if  it  is  a  cutting  tool,  he  is  careful  to  dress  its  edge 
exactly  to  the  job  at  hand  taking  into  account  the  material  to  be 
worked  and  the  character  of  the  shape  to  be  produced.  As  hcmely  as 
this  analogy  may  be,  it  is  singularly  appropriate  for  our  purposes. 

Our  tool  is  a  generalized  one  and  like  a  cutting  tool  blank  is 
useful  only  if  the  proper  edge  is  formed.  This  must  be  accomplished 
by  the  user  and  can  only  be  as  effective  as  the  capability  of  the 
user  to  provide  the  proper  edge.  This  is  accomplished  in  our  general 
expression  through  selection  of  the  measures  selected  for  the  terms 
P  and  U.  This  is  the  critical  criterion  for  successful  use  of  the 
expression. 

Continuing  the  analogy,  the  user  must  ensure  that  successive  use 
of  the  tool  involves  like  material.  The  measures  selected  for  the 
terms  P  &  U  must  be  appropriate  to  the  systems  being  analyzed.  This 
is  apparent  to  the  analyst.  But,  this  is  not  always  so  readily 
apparent  to  successive  review  levels.  There  is  an  inherent  danger  in 
using  indices  in  that  these  may  be  received  on  a  "numbers  are  numbers" 
basis  without  realizing  that  the  numbers  are  to  differing  oases.  To 
avoid  this,  it  becomes  mandatory  that  the  users  of  this  generalized 
expression  explicitly  include  the  index  base  when  citing  numeric 
values  of  E. 

A  second  problem  in  the  use  of  the  expression  evolves  from  the 
fact  that  all  three  terms  are  probabalistic.  This  makes  the  terra  E 
probabalistic  as  it  indeed  is  by  definition.  It  therefore  becomes 
imperative  that  ttv  associated  confidence  factor  be  expressed  when 
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numeric  values  of  E  are  cited.  I  use  the  term  imperative  because 
this  is  an  oft  overlooked  factor.  While  cannon  sense  may  reject  the 
results,  any  statiscal  analyst  can  demonstrate  mathematically  any 
achieved  probability  desired,  from  any  set  of  data,  if  he  is  at 
liberty  to  adjust  the  confidence  factor.  This  factor,  sometimes 
referred  to  as  the  degree  of  goodness,  must  be  expressed  if  valid 
appraisals  are  to  be  made.  This  is  particularly  so  if  these  appraisals 
are  conducted  before  there  is  a  significant  number  of  experience  samples. 

I  would  point  out  here  that  in  many  military  systems,  statistically 
significant  experience  samples  are  never  achieved  except  as  post  mortems. 
It  is  a  fact  of  life  that  in  some  there  may  be  no  one  to  conduct  such 
a  post  mortem. 

I  stated  earlier  that  Systems  Effectiveness  taken  out  of  the 
context  of  Cost  Effectiveness  was  a  sterile  academic  exercise.  Permit 
me  to  set  up  a  hypothetical  situation  to  illustrate  the  point. 


E  =  PAU 

Pa  =  875;  Aa  =  .65;  Ua  =  .7 
Ea  =  875*  .65  *  .7  =  398.125 

Pp  =  300;  Ap  =  .80;  Up  =  .85 
Ep  =  800  *.8o  ‘.85  =  54h.o 

=  875;  Ay  =  .80;  uy  =  .85 
Ey  =  375  •  .80  *.85  =  612.0 

Fig  #2 
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Here  we  have  three  systems  with  a  single  functioned  mission. 

That  is  to  say  that  their  missions  are  identical  and  all  differences 
are  a  direct  function  of  the  systems  themselves. 

System  a  is  determined  to  have  a  performance  index  value  of 
Ta  =  875)  a  calculated  value  of  A^  =  .65  and  sin  estimated  value  of 

Ua  =  .7.  The  value  of  is  398.125. 

System  p  analysis  indicates  values  of  Pp  =  800,  Ap  =  .80  and 
Ug  =  .85.  Thus  Ep  =  5hU.O 

System  y  is  determined  to  have  values  of  =  875 >  Ay  =  .80 
sind  Uy  =  .85.  The  res-  '.tant  Ey  is  612.0. 

Let  us  assume  that  all  the  calculations  are  to  a  common  base 

where  total  mission  capability  is  1000.  For  the  moment,  let  us  also 

assume  that  the  confidence  factors  in  all  three  calculations  are 
the  unlikely  situation  of  being  equivalent. 

It  would  appear  that  system  or  has  a  capability  to  out  perform 
system  3.  In  the  absence  of  system  y,  and  without  a  thorough  analysis 
many  would  elect  it  for  this  reason.  However,  upon  analysis  it  is 
clear  that  system  3  is  superior  on  a  systems  effecti reness  basis 
since  its  E  is  substantially  better  than  system  a . 

But  we  do  have  system  y.  It  combines  the  performance  capability 
of  a  with  the  availability  and  utilisation  characteristics  of  3. 

From  a  systems  effectiveness  point  of  view  its  E  of  612.0  is  clearly 
above  either  3's  5^*0  or  or's  592*125.  In  logic,  since  it  combines 
the  best  of  both  the  other  systems,  as  well  as  in  mathematics,  the 
decision  is  clear.  Support  system  y,  on  the  basis  of  systems  effec¬ 


tiveness. 


The  acquisition  cost  per  system  0 ,  which  employs  state-of-the-art 
circuitry,  is  $1.2  M.  Because  the  majority  of  its  components  are 
proven  and  operator  and  maintenance  personnel  are  familiar  with  both 
the  components  and  the  techniques,  the  utilization  costs  per  system 
are  $3.0  M. 

System  a  also  uses  state-of-the-art  circuitry,  gaining  its 
increased  performance  by  extending  the  components  to  their  limits.  It’s 
acquisition  cost  is  actually  little  more  than  that  of  system  0,  or 
$1.4  M. 

Through  pushing  the  state-of-the-art  by  introducing  new  techniques 
and  components,  system  y  achieves  the  performance  of  system  a  and 
through  redundancy  and  automatic  fault  location  achieves  the  availability 


of  0.  Other  automation  features  overcome  any  negative  effects  of  new 
techniques  and  utilization  matches  0  also.  But  the  acquisition  costs 
including  the  burden  of  development  costs  amount  to  $6  M  per  system. 
The  additional  training  of  personnel,  the  additional  components  to  be 
supported,  the  additional  weight,  space  and  power  requirements  bring 
the  cost  of  utilization  up  to  $8  M. 

From  this  type  of  cost  effectiveness  analysis  it  appears  patent 
that  system  0  buys  more  than  a  by  a  factor  of  two  or  promises  two 
systems  for  each  of  system  a  and  buys  more  than  system  y  by  a  factor 
of  three  -  or  three  for  every  system  y.  This  is  a  n»st  attractive 
situation  -  except  -  what  happens  to  any  forward  progress  or  increase 
in  our  defense  capability? 

Two  important  factors  have  been  overlooked.  One  is  in  the 
systems  effectiveness  calculation  and  the  other  in  the  cost  effective¬ 
ness  expression.  Intelligence  information  indicates  that  the  threat, 
if  it  is  to  be  successfully  countered,  establishes  a  threshold  for  P. 
Anything  less  than  a  value  of  850  will  not  successfully  complete  the 
mission.  As  a  result ,  system  0  rrust  be  rejected  despite  its  dollar 
attractiveness.  It  is  also  conceivable  that  the  term  A  could  have  a 
threshold  value  which  might  reject  system  a.  However,  a  potential 
solution  in  this  case  might  be  to  bring  the  overall  value  of  A  to 
above  threshold  by  using  two  system  redundance.  This  doesn’t  appear 
to  be  attractive  on  a  cost  effectiveness  basis,  however,  because  our 
Ec  for  or  would  become  31.1  as  against  43.7  for  y. 
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between  these  two  expressions  for  Cost  Effectiveness,  and  that  this 

(I)  Ec  =  PAU  Ec  =  W  PAU 

Ca  +  cu  Ca  +  Cu 

was  in  the  term  W,  military  worth. 

In  my  presentation  to  the  Aerospace  Reliability  and  Maintaina¬ 
bility  Symposium,  I  defined  W  as  being 

(II)  W  =  (Fj  Wj  +  ....  Fn  wn) 

n 

where  F  is  the  fraction  of  the  system  effort  devoted  to  a  given  mission 
element  w,  the  summation  of  F}_  +  ....  Fn  being  unity.  This  operative 

was  designed  to  cope  with  the  problem  of  variances  among  multimissioned 
systems  having  a  common  primary  mission  but  with  varying  secondary 
missions.  Prodded  by  Dr.  Hitch  in  his  address  at  the  symposium,  I 
realized  that  my  expression  of  W  was  not  complete.  It  has  long  been 
recognized  that  military  worth  degrades  with  time.  However,  it  is  not 
clear  in  just  what  manner  this  degradation  occurs,  except  chat  it 
varies  as  a  function  of  mission  and  as  a  function  of  the  state-of-the- 
art  —  more  particularly  the  state-of-the-art  as  related  to  potential 
opposition.  It  may  be  linear  in  some  cases,  exponential  in  others  or 
could  be  a  rather  unpredictable  step  function. 

If  we  knew  how  to  quantize  exactly  the  term  W,  the  solution  would 
be  more  difficult  in  that  we  would  then  have  to  quantize  the  degrada¬ 
tion  factor  exactly.  Since  we  do  not,  we  must  express  it  in  terms  of 
a  judgement  index.  This  judgement  index,  as  Dr.  Alain  Enthoven  so 
cogently  pointed  out  in  a  paper  before  the  Ilaval  War  College,  is  not 


purely  intuitive  but  is  premised  on  operational  research  analysis 
coupled  with  experience.  It  would  appear  then  that  a  similar  approach 
to  the  time  degradation  is  no  less  valid.  We  know  intuitively  that 
military  worth  is  real  and  that  it  degrades  with  time.  Indeed  one 
can  readily  envision  a  time  threshold  which  must  be  met  regardless  of 
dollar  costs. 

I  would  therefore  propose  to  introduce  a  new  term  to  the  expression 
to  provide  consideration  of  Time  Effectiveness  (Et)  defined  as  the  index 
of  degradation  of  military  worth  as  a  function  of  time.  The  value  of 
this  index  should  be  established  concomitantly  with  the  index  of 
military  worth  through  gaming  and  other  operational  research  techniques. 
Our  expression  for  Cost  Effectiveness  (dollars)  can  remain  unchanged 
but  our  expression  for  Cost  Effectiveness  or  perhaps  better  identified 
as  Defense  Effectiveness  Ed  becomes 

<;i2)  Ed  =  w_  pau  . 

Et  'Ca  +  Cu' 

Returning  to  cur  hypothetical  case,  were  we  to  assume  linearity 
of  degradation  of  military  worth  with  time,  we  might  arrive  at  a 
scalar  index  with  each  6  month  period  required  to  acquire  being 
weighted  as  one  in  the  index,  we  can  deduce  the  following:  since 
being  identically  missioned  system  or  and  y  can  be  considered  to  have 
equal  values  of  V/  they  can  be  considered  to  have  a  Iv  of  1  and  our 
expression  becomes 
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t  x  y  ^ 

%  =  _k  ,,  pau  > 

Et  'Ca  +  Cu' 

E,  =  1  •  31.1  =  3.65 
9 

Edv  =  1  •  43.7  =  3.24 

lF 

*«-£•  31.1  =  2.2 

Edy  =  1  *  43.7  =  2.4 

IS 

Fig  #4 

thus,  if  for  example  it  would  take  4^  years  to  acquire  system  a  and 
7  years  to  acquire  y,  the  decision  should  appropriately  he  to  elect 
system  or  since  even  buying  two  for  one  the  E^  is  3.65  where.  E^  is 
3.24.  Whereas,  if  a  required  7  years  and  y  9  years,  the  decision 
would  go  to  y  since  =  2.4  and  E^  =  2.2. 

You'll  recall  that  in  my  opening  remarks  I  stated  that  quantizing 
was  conditional  on  when  we  learn  how  to  do  it.  I’m  not  suggesting 
that  I've  learned.  My  intent  is  to  express  a  concept  for  such  an 
approach.  The  purpose  of  this  clinic  is  to  acquaint  you  with  the 
Navy's  needs.  I  used  the  "cutting  tool"  analogy  previously.  I'll 
conclude  with  it.  We  need  to  know  how  to  put  the  proper  edge  on  this 
premising  tool  to  get  a  fine  cut  and  a  better  product  fj^om  our 
efforts  in  Defense  Effectiveness. 
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SYSTEMS  EFFECTIVENESS  -  NAVY 

The  term  "Systems  Effectiveness*  is  plagued,  as  most  popularly-used 
conceptual  terms,  by  a  plethora  of  meanings  In  an  effort  to  avoid  adding 
to  the  confusion  in  this  presentation,  I  3hall  define  my  terms  in  the  context 
which  they  will  be  used  by  me. 

By  "Systems"  I  mean  the  total  complex  of  men  and  machines  required 
to  carry  out  a  military  mission.  These  systems  fall  into  one  or  the  other 
of  two  broad  classes  — "Warfare*  and  "Support"  systems.  These  in  turn  break 
down  into  two  sub-classes:  Warfare  systems.  Active  or  Passive  and  Support 
Systems,  Direct  or  Indirect.  By  way  of  illustration,  Warfare-Active 
would  apply  to  SSBN.  Warfare-Passive  is  illustrated  by  a  mine  defense 
system.  The  OMEGA  navigational  system  is  an  example  of  Support-Direct 
while  SPASUR  is  Support  Indirect  as  is  an  ELINT  system.  The  key  to 
understanding  this  concept  of  system  is  illustrated  by  SSBN  which  includes 
not  only  the  vehicle  but  POLARIS  and  the  various  other  sub-systems  in¬ 
volved  in  performing  the  SSBN  mission  together  with  the  officers  and, men  who 
go  to  make  up  the  SSBN  on  station.  Parenthetically  I  would  add  that  this 
definition  is  not  yet  universally  accepted  in  the  Navy  but  the  trend  is 
in  that  direction. 

The  term  "Systems  Effectiveness*  can  then  be  defined  as  the  probabi¬ 
lity  that  the  system  can  successfully  meet  an  operational  demand  through¬ 
out  a  given  time  period  when  operated  under  specified  conditions. 


This  term  can  be  expressed  as 
(1 )  E  =  PAU 


Where  P  -  Performance  is  a  numerical  index  expressing  system 
capability,  assuming  a  hypothetical  100^  availability,  reliability 
and  utilization  of  performance  capability  in  actual  operation. 

A  -  Availability  is  the  period  or  fraction  of  time  that  the  system 
is  ready  and  capable  of  fully  performing  its  mission  and 

U  -  Utilization  is  the  fraction  of  the  performance  capability 
actually  utilized  due  to  the  specific  application  and  the  environment 
encountered . 

Time  does  not  permit  giving  you  the  complete  rationale  and  deriva¬ 
tion  of  this  expression  or  the  other  mathematical  expressions  which 
I  shall  use.  However,  each  of  you  has  been  provided  with  a  copy  of 
a  paper  entitled  "Systems  Effectiveness-  a  tool  for  appraisal"  which 
does  provide  this  background. 

Continuing  with  the  definitions,  I  would  introduce  the  term  "Cost 
Effectiveness"  since  Systems  Effectiveness  out  of  the  context  of  Cost 
Effectiveness  is  a  sterile  academic  exercise  productive  of  little  but 
to  impress  our  fellows  with  cur  erudition.  Simply  defined,  "Cost 
Effectiveness"  is  the  ratio  between  Systems  Effectiveness  and  its 
attendant  costs.  This  can  be  expressed  as  : 

(2)  Ec  =  W 

Where  W  is  the  index  of  military  worth  of  the  mission  of  the  system 

P,  A,  and  U  are  as  previously  definad. 

Ca  is  the  dollar  cost  of  acquisition  per  system  including  its  pro 
rata  share  of  research  and  development  costs  and 

Cq  is  the  dollar  cost  of  utilization  or  as  it  is  sora  times  referred 


to  cost  of  ownership. 

The  derivation  of  these  latter  two  terms  is  included  in  the  addenda 
sheet  of  the  previously  referenced  paper. 

There  is  yet  another  effectiveness  term  that  I  would  introduce 
which  I  call  Defense  Effectiveness.  This  is  differentiated  from  Cost 
Effectiveness,  which  is  in  terms  of  dollars,  by  the  introduction  of  the 
considerations  of  time  which  is  a  cost  element  too  frequently  overlooked. 
The  term  Defense  Effectiveness  is  probably  best  shown  by  its  expression 
(3)  Ed= 


. W_ /  PAU  \ 

Et  +  y 


Where  all  of  the  other  terras  are  as  previously  defined  and  E^  is 
the  index  of  degradation  of  military  worth  as  a  function  of  time. 

You  will  note  that  both  W  and  E-^  are  indices  which  together  form 
a  co-efficient  with  which  to  numerically  express  the  military  judgment 
factor  which  Dr.  Enthoven  so  clearly  expressed  in  his  paper  before  the 
Naval  War  College,  Newport,  as  being  a  very  necessary  part  of  management 
decisions  in  military  systems.  There  is  admittedly  a  danger  in  the  use 
of  indices  in  that  they  can  be  used  somewhat  indiscriminately  with  grossly 
erroneous  results.  However,  properly  used  they  can  be  very  effective 
tools. 


Ify  purpose  in  taking  you  on  this  little  semantic  excursion  is 
to  give  you  a  feel  for  the  context  in  which  we  in  the  Office  of  Naval 
Material  use  the  term  Systems  Effectiveness.  Only  through  this  feel 
can  one  understand  the  objectives  of  our  systems  effectiveness  effort. 
Before  stating  these,  I  would  give  you  a  bit  more  insight  into  our 
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concept  of  systems  effectiveness  by  citing  what  we  consider  to  be  the 
elements  which  contribute  to  systems  effectiveness. 

These  elements  include  reliability,  maintainability,  compatability, 
operability,  human  factors,  design  simplicity,  logistics  ^upportability, 
etc.  Some  additional  feel  for  this  aspect  of  Systems  Effectiveness  can 
be  obtained  from  the  remarks  of  RADM  E.  A.  Ruckner  at  the  recent  Western 
States  Navy  R&D  Clinic.  Each  of  you  has  been  provided  a  copy  of  his 
remarks . 

Let  us  now  turn  to  the  objectives  of  our  Systems  Effectiveness  Pro¬ 
gram.  In  a  broad  brush  statement,  it  is  to  optimize  Defense  Effectiveness 
in  all  Navy  systems.  To  this  end  we  have  a  number  of  means.  One  of 
these  is  to  develop  methods  for  quantizing  the  elements  of  our  genera¬ 
lized  mathematical  expressions  and  the  elements  of  the  expressions  from 
which  they  were  derived.  This  effort  will  include  the  development  of 
weighting  factors  appropriate  to  the  nature  of  the  mission  of  the  system 
being  evaluated  or  appraised.  For  instance  the  weighting  factors  for 
reliability  ^nd  maintainability  will  not  be  the  same  for  a  manned 
vehicle  as  they  are  for  an  unmanned  vehicle. 

A  second  objective,  in  the  area  of  means,  is  to  apply  the  developed 
numerical  expressions  to  going  systems  projects  to  test  their  validity 
and  refine  them  into  a  useable  management  tool  rather  than  the  conceptual 
expressions  that  they  now  are.  From  this  point,  we  would  apply  them 
as  a  management  tool  for  the  appraisal  of  both  plans  and  projects  in 
execution.  You  will  note  that  I  have  expressed  but  one  end.  The  others 
are  means.  This  shall  be  so  for  the  remainder  of  this  presentation. 


We  in  ONM  have  but  one  objective  which  ia  an  end  and  that  is  to  provide 
a  maximum  Defense  Effectiveness  to  the  Navy.  We  will  NOT  strive  for 
reliability  for  reliability ' 3  sake,  maintainability  for  maintainability's 
sake  or  even  low  dollar  costs  solely  for  the  sake  of  low  dollar  costs. 

All  of  the  elements,  even  including  Performance  Capability,  are  tradeable 
items  which  can  be  supported  ONLY  insofar  as  they  contribute  to  the 
maximization  of  the  Defense  Effectiveness  index  of  each  system. 

Since  these  items  are  tradeable,  we  must  have  a  medium  of  exchange 
for  evaluating  our  tradeoffs.  Numbers  to  assign  to  these  elements  there¬ 
fore  become  imperative.  They  are  needed  not  only  as  a  method  for 
appraisal  in  our  own  thinking  but  also  as  a  medium  for  communicating 
our  appraisals  and  management  rationale  to  others.  It  is  the  recogni¬ 
tion  of  this  that  has  motivated  us  to  attempt  a  generalized  mathematical 
expression  to  provide  a  conceptual  frame  work  within  which  we  could  examine 
the  principal  elements  of  performance,  dollar  costs,  time  and  military 
worth  and  play  them  off  one  against  the  other  to  the  end  of  maximizing 
Defense  Effectiveness. 

We  in  the  Navy,  have  understood  for  some  time  that  mathematical 
modeling  is  a  powerful  analytical  tool.  Our  operational  research  people 
have  used  it  extensively.  Gaming  and  cueing  theories  have  permitted 
examination  of  both  strategic  and  tactical  concepts  without  the  tremen¬ 
dous  expense  of  extensive  exercises  of  ships,  submarines  and  aircraft. 
Heretofore,  we  have  lacked  a  generalized  model  with  which  to  game  our 
development  and  design  concepts.  The  use  of  specific  models  has  been 
attempted  in  the  astronautics  field  with  very  promising  results. 


NASA  particularly  has  been  very  successful  in  modeling  their  manned 
space  systems.  This  kind  of  analysis,  while  expensive,  is  many  orders 
of  magnitude  less  expensive  in  both  dollars  and  time  than  the  "let's 
build  one  and  try  it"  approach.  It  is  felt  that  NASA's  fine  record  of 
successful  flights  is  attributable  in  no  small  measure  to  the  careful 
analysis  and  mathematical  modeling  work  done  prior  to  hardware  commit¬ 
ments.  Indeed  much  of  the  Navy's  success  in  POLARIS  stems  from  the 
same  kind  of  approach.  Recognizing  this  we  have  pressed  forward  to 
achieve  a  means  for  applying  this  tool  generally  in  Naval  Development. 

Who  is  this  "we"  that  I've  alluded  to?  What  is  the  organization 
to  implement  this  new  approach  in  the  Navy  to  Systems  Effectiveness? 

At  present,  the  overall  organization  is  dominantly  informal.  However, 
there  are  formally  organized  pockets  in  various  areas.  The  focal  point 
in  the  Navy  of  the  informal  organization  is  the  Systems  Effectivene-s 
Branch  of  the  Systems  Development  Division  of  the  Office  of  Naval  Material. 
Under  the  leadership  of  RADM  E.  A  Ruckner,  the  Deputy  Chief  of  Naval 
Material  for  Development  and  the  Chief  of  Naval  Development  and  armed 
with  a  charter  granted  by  VADM  W.  A.  Scboech,  the  Chief  of  Naval  Material, 
this  s.aall,  tightly-knit  group  has  the  responsibility  for  developing  the 
policies  and  procedures  to  insure  that  Systems  Effectiveness  is  achieved 
by  all  the  segments  of  the  Naval  Material  Support  Establishment  which  en¬ 
compasses  the  four  material  bureaus  and  their  field  activities.  Further 
this  group  reviews  for  Systems  Effectiveness  assurance  all  Techrical 
Development  Plans  arid  Proposed  Technical  Approaches  before  they  are 
submitted  to  the  Chief  of  Naval  Operations  and  the  Secretarial  levels. 
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At  tl^is  juncture  I  would  point  out  that  a  number  of  these  documents, 
without  which  funds  will  not  be  allocated  by  the  Office  of  the  Director 
of  Defense  Research  and  Engineering,  have  been  returned  to  their 
originators  for  rework  because  the  Systems  Effectiveness  assurance 
part  of  the  plans  was  not  deemed- adequate.  This  group  is  further 
supported  by  ghiter  levels  in  the  larger  systems  by  DOD  Directive 
3200.9  which  established  Project  Definition  Phase  for  large  systems. 

Each  of  you  has  been  provided  a  copy  of  this  directive  together  with 
the  Navy  Implementing  Instruction  which  was  written  within  this  group. 

I  alluded  earlier  to  organized  pockets.  Let  me  cite  some  of  them. 
Rather  than  tc  refer  to  the  somewhat  confusing  code  structure  of  the 
cureaus'  organizations,  I’ll  refer  u  them  by  project  names.  Again 
in  the  interest  of  time  in  this  presentation,  each  of  you  has  received 
a  brochure  for  each  of  these  projects  from  which  you  can  obtain  a 
greated  depth  of  understanding  of  each  of  these  projects.  Lest  I 
be  accused  of  bias,  I’ll  cite  the  projects  in  alphabetical  order. 

METRI  "Military  Essentiality  Through  Readiness  Indices"  is  a 
project  under  the  Bureau  of  Supplies  ?nd  Accounts.  This  is  a  project 
for  relating  readiness  of  the  fleet  with  its  supporting  items  in 
simulated  form  to  provide  important  decision-making  information  on 
problems  of  military  essentiality  and  readiness.  The  ultimate  objective 
is  to  get  intelligence  regarding;  (a)  the  readiness  of  force  <mits 
at  any  one  time,  (b)  how  readiness  might  be  improved  and  (c)  the  ^x+ent 
that  individual  components  affect  readiness.  Here  we  have  in  development 
one  aspect  of  measurement  to  permit  quantizing  the  elements  of  our 


generalize  mathematical  expression  of  Effectiveness. 

PACED  "Program  for  Advanced  Concepts  in  Electorates  Design"  is  a 
project  under  the  Bureau  of  Ships  with  the  Naval  Applied  Science  Labora¬ 
tory  as  the  lead-laboratory.  The  thrust  of  this  project  is  to  develop 
measurement  techniques  and  methodology  for  System  Effectiveness  analysis 
and  appraised  with  immediate  reference  to  electronics  systems  and 
subsequent  extrapolation  to  non-electronic  syst  ms. 

SEAHAWK  -  The  Advanced  Design  AS W  Destroyer.  This  is  a  joint 
project  between  the  Bureau  of  Ships  and  the  Bureau  of  Naval  Weapons. 

The  thrust  of  the  concept  underlying  this  effort  is  the  design  of  a 
total  ship  as  an  integrated  system  rather  than  the  collecting  together 
of  a  number  of  so-called  systems  into  a  common  envelope  called  a  ship. 
SEAHAWK  together  with  a  somewhat  similar  approach  in  submarines  called 
FRISCO  provide  the  vehicles  for  the  PACED  effort.  These  are  an  indica¬ 
tion  of  the  trend  toward  the  larger  concept  of  "System"  that  I  alluded 
to  earlier.  SEAHAWK  management  is  unique  in  that  a  voluntary  joint 
BUSHIPS/BUWEPS  Project  Management  Office  was  established  to  implement  the 
system  design  concept  well  before  the  reorganization  of  the  Navy  Depart¬ 
ment. 

VAST  -  "Versatile  Avionics  Shop  Test"  System  is  a  Bureau  of  Naval 
Weapons  project.  This  may  be  of  somewhat  more  interest  to  the  maintain¬ 
ability  oriented  members  of  the  class.  The  VAST  project  addresses  it¬ 
self  to  the  development  of  a  standardized  Avionic  Shop  for  Aircraft  Carriers 
having  broad  applicability  to  aircraft  types  with  simplified  operations 
and  maintenance  of  the  test  set-up  and  reduced  "turn-around"  time  for 


the  avionics  portion  of  the  aircraft. 

These  are  six  areas  in  the  NMSE  where  organized  efforts  are  being 
made  and  projects  are  going  forward  which  have  as  their  central  theme 
the  achievement  of  systems  effectiveness.  I  alluded  to  them  as  pockets. 
They  are  pockets  in  the  sense  that  they  are  organizationally  isolated 
one  from  the  other  and,  prior  to  the  establishment  of  the  Systems 
Effectiveness  Branch  in  ONM,  had  no  focal  point  for  their  efforts. 

They  are  not  the  only  pockets.  There  are  others. 

I  would  remiss,  if  I  confined  myself  to  the  NMSE  alone.  One  very 
significant  Navy  effort  toward  achieving  Systems  Effectiveness  is  taking 
place  outside  of  the  NMSE.  This  is  the  program  of  the  Office  of  the 
Chief  of  Naval  Personnel  referred  to  as  the  New  Developments  Human  Factors 
Program.  While  this  most  important  program,  as  indicated  in  the  paper 
by  Mr.  William  Hopkins  which  has  been  provided  each  of  you,  is  being 
prosecuted  by  the  SOPERS  organization,  it  is  not  entirely  outside  of 
our  Systems  Effectiveness  Group.  As  a  part  of  our  staff  we  have  a  full¬ 
time  liaison  officer  from  BuPers.  Further,  BuPers  Human  Factors  teams 
are  located  within  both  the  Bureau  of  Ships  and  the  Bureau  of  Naval 
Weapons. 

I  would  hope  that  none  of  you  have  any  questions  in  your  minds  as 
to  why  I  have  included  this  effort  in  a  discussion  of  Systems  Effective¬ 
ness.  If  perchance  you  do  let  me  explain.  We  are  and  have  been  living 
in  an  age  of  man-machine  systems.  Therefore  man-effectiveness  is 
every  bit  as  important  as- machine  effectiveness.  Moreover,  we  must 
achieve  a  blending  of  the  two  in  order  to  achieve  maximum  system 
effectiveness  through  optimum  utilization  of  man  vis-a-vis  the  machine 


and  vice  versa. 


lou  are  going  to  see  a  greatly  intensified  emphasis  of  this  in  the 
very  net"  oiure.  We  are  quite  aware  that  man  reliability,  man  maintain¬ 
ability;  men  operability,  and  man  supportability  are  explicitly  factors 
which  contribute  to  our  generalized  mathematical  expression.  As  a  matter 
of  fact,  our  Systems  Effectiveness  Group  has  an  Experimental  Psychologist 
as  a  member  in  addition  to  the  BuPers  Liaison  Officer,  We  in  the  NMSE 
do  understand  our  Human  Factors  Engineering  responsibilities  in  develop¬ 
ment.  By  "we*  in  this  case  I  include  our  top  people.  By  way  of  illustra¬ 
tion,  there  is  more  than  a  little  evidence  that  my  selection  as  the  Systems 
Effectiveness  Branch  Head  was  premised  in  large  measure  upon  my  training 
in  psychology  in  addition  to  my  engineering  experience. 

With  the  establishment  of  the  Systems  Effectiveness  Branch  about 
six  months  age,  there  is  a  focal  point  for  the  informal  organization.  It 
is  our  intention  to  provide  a  cohesiveness  to  the  Navy' 3  Systems  Effective¬ 
ness  effort  and  to  insure  maximum  cross-pollination  among  the  various 
projects. 

Somewhat  inadequate  staffing  has  hampered  these  efforts.  The 
staffing  situation  stems  from  two  sources.  First,  the  usual  problem 
of  getting  ceiling  points  allocated,  in  a  climate  in  Washington  of  reducing 
numbers  at  the  seat  of  the  government,  has  been  a  factor.  Perhaps  the 
more  significant  ’actor  has  been  the  difficulty  of  finding  the  ca?  Yoer 
of  personnel  which  we  feel  is  required  to  do  the  Aob  properly.  Theve  are 
far  too  few  cf  this  type  of  people  anywhere  and  many  of  them  are  either 
disenchanted  with  the  rather  frantic  Washington  working  environment  or 
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with  the  Civil  Service  pay  scales.  Nevertheless,  under  the  persuasion 
that  it  is  best  to  make  haste  slowly  we  feel  confident  that  we'll  resolve 
that  staffing  problem. 

Our  greater  problem  is  one  of  education.  This  has  several  aspects. 

As  a  matter  of  fact,  I  am  working  at  the  resolution  of  one  of  them  at 
this  very  moment.  Too  few  working  designers  within  and  without  government 
understand  what  we  are  trying  to  do  and  what  their  role  is  in  relation 
to  the  whole.  When  one  realizes  just  how  many  designers  and  engineers  con¬ 
tribute  to  just  one  ship,  cne  begins  to  appreciate  the  magnitude  of  the 
educational  task  ahead  of  us.  The  concepts  which  I  expressed  at  the 
beginning  of  this  presentation  have  not  been  taught  in  the  Universities 
and  generally  are  still  not  being  taught.  Further,  Systems  Effectiveness 
involves  a  degree  of  probabilistic  thinking  which  only  the  very  junior 
engineers  have  been  appreciably  exposed  to  in  their  education.  Additionally, 
the  probalistic  reasoning  involved  in  Systems  Effectiveness  is  in  a  sense 
abhorent  to  the  deterministic  exactness  which  is  the  hallmark  of  the 
engineering  profession. 

But  our  educational  problems  are  not  all  downward.  Looking  up 
we  find  people  who  are  quite  capable  at  probablistic  thanking  at  the 
poker  tables,  the  races  or  their  brokers.  Yet  they  have  a  moralistic 
block  e gainst  extrapolating  this  kind  of  thinking  to  the  job.  The 
stipaa  of  gambling  concepts  applied  to  thetr  sense  of  responsibility  is 
not  tolerable.  Yet — paradoxically — they  glibly  ude  the  term  "calculated 
risk".  Perhaps  if  we  can  persuade  them  that  we  are  trying  to  put  real 
calculation  into  this  term,  we  will  go  a  long  way  in  this  educational  effort. 
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To  the  solution  of  our  educational  problem,  we,  in  the  ONM  Systems 
Effectiveness  Group,  are  taking  every  opportunity  to  educate  through 
professional  symposia,  courses  such  as  this  and  the  issuance  of  specifi¬ 
cation  criteria  and  guidance  handbooks.  In  your  packets  you  have  three 
such  documents.  BuVeps  specifications  VS  3250  and  WR-30  and  BuShips 
specification  M3L-R-22732B ( SHIPS).  All  three  of  these  are  in  need  of 
updating  and — to  some  extent — clarification.  But  they  are  indicative  of 
the  work  being  done;  In  the  Systems  Effectiveness  Branch  ve  have  a 
Technical  Development  Plan  Handbook  which  ve  expect  to  promulgate  later 
this  calendar  year  which  is  Intended  to  provide  education  and  guidance 
in  this  area.  NAVXAT  NOTICE  3960,  a  copy  of  which  has  been  provided  to 
you,  is  an  in-house  attempt  to  provide  education  and  guidance.  This  kind 
of  documentation  is  most  difficult  to  prepare.  When  dealing  with  concepts, 
the  semantics  problems  of  the  written  word  become  exterme.  This  is 
worsened  when  they  are  received  with  something  less  than  complete  enthusiasm 
by  reason  of  their  appearing  to  require  additional  work.  And,  indeed  they 
do,  at  the  outset.  Homework  has  never  been  very  attractive.  In  essence, 
we  are  requiring  additional  homework.  The  pay  off,  of  course,  is  the  final 
grade.  —  And,  I  might  add,  a  reduction  in  cramming.  Nevertheless,  we 
have  experienced  many  situations  which  oako  it  questionable  whether  or 
not  people  are  truly  trying  to  understand. 

An  addition  to  our  educational  problem  is  the  fact  of  life  situation 
that  the  operators  are  experiencing  parallel  difficulties  in  attempting  to 
quantise  their  requirements  so  that  a  mathematical  framework  exists  for  the 
evaluation  of  Sjysteas  Effectiveness.  Several  terms  in  our  Defense  Effective¬ 
ness  expression 
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must  come  from  the  operators  In  their  setting  forth  the  operational  require¬ 
ment.  The  terns  W  &  Et  must  come  from  them.  Threshold  values  for  P 
and  A  must  be  established  premised  upon  quantitative  factors  in  the 
operational  requirement.  These  have  to  be  communicated  through  threat 
analysis  and  evaluation.  Even  as  we  have  much  to  learn  in  quantizing 
our  factors,  the  operational  researchers  are  still  groping  for  valid 
measures  of  their  factors. 

Despite  these  shortcomings  in  exactness,  there  is  much  that  can  be 
done.  This  is  demonstrated  by  the  projects  I've  cited.  The  house  built 
out  of  rough  hewn  lumber  offers  a  great  deal  of  protection  from  the  storm. 
We'd  be  stupid  to  stay  out  in  the  rain  simply  because  our  house  isn't 
entirely  leak  proof.  As  we  get  better  tools  to  work  with,  we'll  get 
better  fit  and  fewer  leaks.  In  the  meantime,  we'll  use  the  tools  we  have 
to  get  on  with  the  job,  and  at  the  same  time  work  at  getting  a  better  edge 
on  our  tools. 
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